Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240506 19:56:28.361961 12483 test_util.cc:268] Using random seed: 1997396156
2024-05-06T19:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T19:56:28Z Disabled control of system clock
I20240506 19:56:28.396999 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.254:33495
--webserver_interface=127.12.48.254
--webserver_port=0
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 19:56:28.733727 12491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:28.734089 12491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:28.734228 12491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:56:28.741892 12491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 19:56:28.742038 12491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:28.742125 12491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 19:56:28.742218 12491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 19:56:28.751487 12491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.254:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.48.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:28.752928 12491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:28.754837 12491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 19:56:28.767112 12496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:28.767683 12497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:28.768288 12499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:28.769114 12491 server_base.cc:1034] running on GCE node
I20240506 19:56:28.769980 12491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:28.771562 12491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:28.772795 12491 hybrid_clock.cc:648] HybridClock initialized: now 1715025388772745 us; error 36 us; skew 500 ppm
I20240506 19:56:28.773159 12491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:28.780438 12491 webserver.cc:460] Webserver started at http://127.12.48.254:40249/ using document root <none> and password file <none>
I20240506 19:56:28.781205 12491 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:28.781361 12491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:28.781697 12491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 19:56:28.784552 12491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/instance:
uuid: "190a2eabb30d4154960a44eb50a8a87d"
format_stamp: "Formatted at 2024-05-06 19:56:28 on dist-test-slave-gjg4"
I20240506 19:56:28.785355 12491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal/instance:
uuid: "190a2eabb30d4154960a44eb50a8a87d"
format_stamp: "Formatted at 2024-05-06 19:56:28 on dist-test-slave-gjg4"
I20240506 19:56:28.791236 12491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240506 19:56:28.795020 12505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:28.796514 12491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240506 19:56:28.796703 12491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
uuid: "190a2eabb30d4154960a44eb50a8a87d"
format_stamp: "Formatted at 2024-05-06 19:56:28 on dist-test-slave-gjg4"
I20240506 19:56:28.796926 12491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:28.853170 12491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:28.854406 12491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:28.854740 12491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:28.872745 12491 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.254:33495
I20240506 19:56:28.872762 12556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.254:33495 every 8 connection(s)
I20240506 19:56:28.874207 12491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/info.pb
I20240506 19:56:28.875936 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 12491
I20240506 19:56:28.876195 12483 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal/instance
I20240506 19:56:28.878270 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:28.878754 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.253:36819
--webserver_interface=127.12.48.253
--webserver_port=0
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 19:56:28.884948 12557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
I20240506 19:56:28.901046 12557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
W20240506 19:56:28.902263 12508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111)
W20240506 19:56:28.903776 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111)
I20240506 19:56:28.953116 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } attempt: 1
W20240506 19:56:28.955296 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111)
I20240506 19:56:29.023558 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } attempt: 2
W20240506 19:56:29.025944 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111)
I20240506 19:56:29.117210 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } attempt: 3
W20240506 19:56:29.119266 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111)
W20240506 19:56:29.225023 12559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:29.225415 12559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:29.225553 12559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:56:29.233273 12559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 19:56:29.233433 12559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:29.233517 12559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 19:56:29.233601 12559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 19:56:29.244695 12559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.253:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.48.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:29.246217 12559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:29.248114 12559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 19:56:29.259824 12567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:29.260596 12570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:29.261188 12568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:29.261976 12559 server_base.cc:1034] running on GCE node
I20240506 19:56:29.262514 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } attempt: 4
I20240506 19:56:29.262991 12559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20240506 19:56:29.264317 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111)
I20240506 19:56:29.264611 12559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:29.265859 12559 hybrid_clock.cc:648] HybridClock initialized: now 1715025389265791 us; error 31 us; skew 500 ppm
I20240506 19:56:29.266239 12559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:29.268718 12559 webserver.cc:460] Webserver started at http://127.12.48.253:35887/ using document root <none> and password file <none>
I20240506 19:56:29.269738 12559 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:29.269914 12559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:29.270304 12559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 19:56:29.273778 12559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/instance:
uuid: "27d4482cd59a4e5687992b1f8a177373"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
I20240506 19:56:29.274593 12559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal/instance:
uuid: "27d4482cd59a4e5687992b1f8a177373"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
I20240506 19:56:29.280288 12559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240506 19:56:29.283876 12576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:29.285234 12559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:29.285452 12559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
uuid: "27d4482cd59a4e5687992b1f8a177373"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
I20240506 19:56:29.285651 12559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:29.303932 12559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:29.304821 12559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:29.305126 12559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:29.327142 12559 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.253:36819
I20240506 19:56:29.327145 12627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.253:36819 every 8 connection(s)
I20240506 19:56:29.328616 12559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/info.pb
I20240506 19:56:29.329789 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 12559
I20240506 19:56:29.329990 12483 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal/instance
I20240506 19:56:29.331423 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.252:36867
--webserver_interface=127.12.48.252
--webserver_port=0
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 19:56:29.333673 12628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:29.339954 12628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
I20240506 19:56:29.353634 12628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20240506 19:56:29.362717 12628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } has no permanent_uuid. Determining permanent_uuid...
W20240506 19:56:29.363814 12578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
W20240506 19:56:29.365365 12628 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
I20240506 19:56:29.414664 12628 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 1
W20240506 19:56:29.416841 12628 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
I20240506 19:56:29.485090 12628 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 2
W20240506 19:56:29.487145 12628 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
I20240506 19:56:29.563580 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } attempt: 5
I20240506 19:56:29.570257 12557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } has no permanent_uuid. Determining permanent_uuid...
W20240506 19:56:29.572499 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
I20240506 19:56:29.578368 12628 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 3
W20240506 19:56:29.580394 12628 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
I20240506 19:56:29.623775 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 1
W20240506 19:56:29.625964 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
W20240506 19:56:29.668812 12630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:29.669131 12630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:29.669255 12630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:56:29.676870 12630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 19:56:29.676999 12630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:29.677055 12630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 19:56:29.677158 12630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 19:56:29.686565 12630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.252:36867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.48.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:29.688006 12630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:29.689888 12630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 19:56:29.694200 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 2
W20240506 19:56:29.696597 12557 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
W20240506 19:56:29.701689 12641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:29.702293 12639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:29.702927 12638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:29.703656 12630 server_base.cc:1034] running on GCE node
I20240506 19:56:29.704463 12630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:29.705965 12630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:29.707197 12630 hybrid_clock.cc:648] HybridClock initialized: now 1715025389707121 us; error 56 us; skew 500 ppm
I20240506 19:56:29.707544 12630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:29.709968 12630 webserver.cc:460] Webserver started at http://127.12.48.252:37261/ using document root <none> and password file <none>
I20240506 19:56:29.710692 12630 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:29.710824 12630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:29.711138 12630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 19:56:29.713779 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/instance:
uuid: "97f390a786034cddbff757779037f861"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
I20240506 19:56:29.714493 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal/instance:
uuid: "97f390a786034cddbff757779037f861"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
I20240506 19:56:29.719631 12630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 19:56:29.723183 12647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:29.723676 12628 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 4
I20240506 19:56:29.724965 12630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:29.725165 12630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
uuid: "97f390a786034cddbff757779037f861"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
W20240506 19:56:29.725668 12628 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.12.48.252:36867: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111)
I20240506 19:56:29.725543 12630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:29.742062 12630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:29.742941 12630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:29.743242 12630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:29.762167 12630 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.252:36867
I20240506 19:56:29.762187 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.252:36867 every 8 connection(s)
I20240506 19:56:29.763615 12630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/info.pb
I20240506 19:56:29.767374 12699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:29.769556 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 12630
I20240506 19:56:29.769758 12483 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal/instance
I20240506 19:56:29.773366 12699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
I20240506 19:56:29.788404 12699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20240506 19:56:29.795795 12699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } has no permanent_uuid. Determining permanent_uuid...
I20240506 19:56:29.802826 12557 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 3
I20240506 19:56:29.813078 12699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Bootstrap starting.
I20240506 19:56:29.815342 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Bootstrap starting.
I20240506 19:56:29.817137 12699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:29.818298 12699 log.cc:824] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:29.819028 12557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:29.820559 12557 log.cc:824] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:29.821985 12699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: No bootstrap required, opened a new log
I20240506 19:56:29.823462 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: No bootstrap required, opened a new log
I20240506 19:56:29.831578 12699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:29.832022 12699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:29.832167 12699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f390a786034cddbff757779037f861, State: Initialized, Role: FOLLOWER
I20240506 19:56:29.832783 12557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:29.832927 12699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:29.833141 12557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:29.833277 12557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190a2eabb30d4154960a44eb50a8a87d, State: Initialized, Role: FOLLOWER
I20240506 19:56:29.834018 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:29.834084 12557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:29.834285 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:29.835121 12711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:29.835418 12699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 19:56:29.835577 12711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:29.836284 12557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: configured and running, proceeding with master startup.
I20240506 19:56:29.842254 12711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:29.842521 12711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:29.844302 12711 leader_election.cc:288] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:29.845250 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "190a2eabb30d4154960a44eb50a8a87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:29.845933 12674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190a2eabb30d4154960a44eb50a8a87d in term 0.
I20240506 19:56:29.846534 12507 leader_election.cc:302] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190a2eabb30d4154960a44eb50a8a87d, 97f390a786034cddbff757779037f861; no voters:
I20240506 19:56:29.847072 12711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:29.847213 12711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:29.847309 12711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:29.850060 12711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:29.851585 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "190a2eabb30d4154960a44eb50a8a87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861"
I20240506 19:56:29.851809 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:29.847813 12491 master_runner.cc:186] Error getting master registration for 127.12.48.253:36819: OK, instance_id { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" instance_seqno: 1715025389321077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20240506 19:56:29.857828 12511 tablet.cc:2369] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:56:29.858732 12674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190a2eabb30d4154960a44eb50a8a87d in term 1.
I20240506 19:56:29.859445 12507 leader_election.cc:302] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190a2eabb30d4154960a44eb50a8a87d, 97f390a786034cddbff757779037f861; no voters:
W20240506 19:56:29.861919 12730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 19:56:29.862056 12730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 19:56:29.859548 12630 master_runner.cc:186] Error getting master registration for 127.12.48.253:36819: OK, instance_id { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" instance_seqno: 1715025389321077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 19:56:29.868242 12630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 19:56:29.858004 12711 leader_election.cc:288] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [CANDIDATE]: Term 1 election: Requested vote from peers 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:29.868666 12711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:29.868954 12711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [term 1 LEADER]: Becoming Leader. State: Replica: 190a2eabb30d4154960a44eb50a8a87d, State: Running, Role: LEADER
I20240506 19:56:29.869630 12711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
W20240506 19:56:29.872570 12733 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 19:56:29.872717 12733 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 19:56:29.881847 12491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 19:56:29.890611 12736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a2eabb30d4154960a44eb50a8a87d. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:29.891096 12736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:29.898190 12738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 19:56:29.900883 12738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 19:56:29.912971 12674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Refusing update from remote peer 190a2eabb30d4154960a44eb50a8a87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:29.914152 12736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:29.937690 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a2eabb30d4154960a44eb50a8a87d. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:29.938076 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:29.941952 12739 mvcc.cc:205] Tried to move back new op lower bound from 7024743997070905344 to 7024743996911681536. Current Snapshot: MvccSnapshot[applied={T|T < 7024743997070905344}]
I20240506 19:56:29.942804 12736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:29.943135 12736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:29.944866 12738 catalog_manager.cc:1347] Generated new cluster ID: cab80095522b47ec9cab71d12c4cac98
I20240506 19:56:29.944988 12738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 19:56:29.946192 12740 mvcc.cc:205] Tried to move back new op lower bound from 7024743997070905344 to 7024743996911681536. Current Snapshot: MvccSnapshot[applied={T|T < 7024743997070905344}]
I20240506 19:56:29.969980 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:29.970299 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:29.981810 12738 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 19:56:29.987488 12738 catalog_manager.cc:1504] Loading token signing keys...
I20240506 19:56:30.019438 12738 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Generated new TSK 0
I20240506 19:56:30.022469 12738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 19:56:30.024885 12628 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } attempt: 5
I20240506 19:56:30.039355 12628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Bootstrap starting.
I20240506 19:56:30.046262 12628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:30.048032 12628 log.cc:824] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:30.053551 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.48.193:0
--local_ip_for_outbound_sockets=127.12.48.193
--webserver_interface=127.12.48.193
--webserver_port=0
--tserver_master_addrs=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240506 19:56:30.061643 12628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: No bootstrap required, opened a new log
I20240506 19:56:30.072929 12628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:30.073563 12628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:30.073774 12628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d4482cd59a4e5687992b1f8a177373, State: Initialized, Role: FOLLOWER
I20240506 19:56:30.074779 12628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:30.081993 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:30.084599 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:30.084471 12602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "190a2eabb30d4154960a44eb50a8a87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373"
I20240506 19:56:30.084416 12628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 19:56:30.085069 12602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:30.085769 12603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "190a2eabb30d4154960a44eb50a8a87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
I20240506 19:56:30.091441 12602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190a2eabb30d4154960a44eb50a8a87d in term 1.
I20240506 19:56:30.092249 12601 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Refusing update from remote peer 190a2eabb30d4154960a44eb50a8a87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:30.094408 12711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:30.153352 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a2eabb30d4154960a44eb50a8a87d. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:30.153693 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:30.154191 12755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:30.154706 12755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:30.161631 12763 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20240506 19:56:30.575831 12745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:30.576148 12745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:30.576298 12745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:56:30.584220 12745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:30.584471 12745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.48.193
I20240506 19:56:30.594076 12745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.48.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.48.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.48.193
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:30.595551 12745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:30.597445 12745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 19:56:30.597900 12755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:30.598110 12755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:30.599463 12755 leader_election.cc:288] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:30.600073 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:30.600239 12532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:30.601100 12580 leader_election.cc:302] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27d4482cd59a4e5687992b1f8a177373; no voters: 190a2eabb30d4154960a44eb50a8a87d, 97f390a786034cddbff757779037f861
I20240506 19:56:30.601552 12755 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 19:56:30.611799 12769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:30.612287 12768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:30.612891 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:30.613613 12745 server_base.cc:1034] running on GCE node
I20240506 19:56:30.614431 12745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:30.614967 12711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:30.615201 12711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:30.615897 12745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:30.617238 12745 hybrid_clock.cc:648] HybridClock initialized: now 1715025390617118 us; error 77 us; skew 500 ppm
I20240506 19:56:30.617599 12745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:30.620255 12745 webserver.cc:460] Webserver started at http://127.12.48.193:41115/ using document root <none> and password file <none>
I20240506 19:56:30.621024 12745 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:30.621162 12745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:30.621515 12745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 19:56:30.624392 12745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data/instance:
uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
format_stamp: "Formatted at 2024-05-06 19:56:30 on dist-test-slave-gjg4"
I20240506 19:56:30.625504 12745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/instance:
uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
format_stamp: "Formatted at 2024-05-06 19:56:30 on dist-test-slave-gjg4"
I20240506 19:56:30.631461 12745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20240506 19:56:30.635411 12780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:30.637224 12745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:30.637475 12745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal
uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
format_stamp: "Formatted at 2024-05-06 19:56:30 on dist-test-slave-gjg4"
I20240506 19:56:30.637696 12745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:30.654875 12745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:30.655769 12745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:30.656091 12745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:30.657111 12745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 19:56:30.658825 12745 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 19:56:30.658902 12745 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:30.659014 12745 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 19:56:30.659075 12745 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:30.692616 12745 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.193:39475
I20240506 19:56:30.692662 12892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.193:39475 every 8 connection(s)
I20240506 19:56:30.693961 12745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/data/info.pb
I20240506 19:56:30.696753 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 12745
I20240506 19:56:30.696969 12483 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/instance
I20240506 19:56:30.701987 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.48.194:0
--local_ip_for_outbound_sockets=127.12.48.194
--webserver_interface=127.12.48.194
--webserver_port=0
--tserver_master_addrs=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240506 19:56:30.721580 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:30.722002 12895 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:30.722007 12893 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:30.722205 12895 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:30.723488 12893 heartbeater.cc:511] Master 127.12.48.252:36867 requested a full tablet report, sending...
I20240506 19:56:30.723527 12895 heartbeater.cc:511] Master 127.12.48.253:36819 requested a full tablet report, sending...
I20240506 19:56:30.724812 12894 heartbeater.cc:347] Connected to a master server at 127.12.48.254:33495
I20240506 19:56:30.724995 12894 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:30.725695 12894 heartbeater.cc:511] Master 127.12.48.254:33495 requested a full tablet report, sending...
I20240506 19:56:30.726305 12664 ts_manager.cc:194] Registered new tserver with Master: ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:30.727546 12521 ts_manager.cc:194] Registered new tserver with Master: ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:30.727978 12592 ts_manager.cc:194] Registered new tserver with Master: ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:30.729858 12521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.48.193:41395
I20240506 19:56:30.863955 12730 catalog_manager.cc:1259] Loaded cluster ID: cab80095522b47ec9cab71d12c4cac98
I20240506 19:56:30.864102 12730 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: loading cluster ID for follower catalog manager: success
I20240506 19:56:30.867597 12730 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: acquiring CA information for follower catalog manager: success
I20240506 19:56:30.869657 12730 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 19:56:30.918699 12904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:30.918922 12904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:30.920418 12904 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:56:30.921025 12532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:30.921226 12601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
I20240506 19:56:30.921942 12651 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:56:30.922449 12904 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 19:56:31.063203 12903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:31.063532 12903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:31.063678 12903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:56:31.071579 12903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:31.071830 12903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.48.194
I20240506 19:56:31.081948 12903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.48.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.48.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.48.194
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:31.083678 12903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:31.085760 12903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 19:56:31.100579 12909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:31.101310 12910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:31.102293 12912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:31.102995 12903 server_base.cc:1034] running on GCE node
I20240506 19:56:31.103917 12903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:31.105168 12903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:31.106552 12903 hybrid_clock.cc:648] HybridClock initialized: now 1715025391106463 us; error 67 us; skew 500 ppm
I20240506 19:56:31.106978 12903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:31.109874 12903 webserver.cc:460] Webserver started at http://127.12.48.194:42781/ using document root <none> and password file <none>
I20240506 19:56:31.110661 12903 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:31.110803 12903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:31.111121 12903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 19:56:31.113943 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data/instance:
uuid: "059a5c4a3f7e49ef95af4b8341702745"
format_stamp: "Formatted at 2024-05-06 19:56:31 on dist-test-slave-gjg4"
I20240506 19:56:31.114689 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/instance:
uuid: "059a5c4a3f7e49ef95af4b8341702745"
format_stamp: "Formatted at 2024-05-06 19:56:31 on dist-test-slave-gjg4"
I20240506 19:56:31.117061 12917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:31.117240 12917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:31.118024 12917 leader_election.cc:288] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:31.118638 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:31.118654 12532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:31.119405 12580 leader_election.cc:302] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27d4482cd59a4e5687992b1f8a177373; no voters: 190a2eabb30d4154960a44eb50a8a87d, 97f390a786034cddbff757779037f861
I20240506 19:56:31.119751 12917 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:31.120862 12903 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240506 19:56:31.124528 12919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:31.126189 12903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 19:56:31.126358 12903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal
uuid: "059a5c4a3f7e49ef95af4b8341702745"
format_stamp: "Formatted at 2024-05-06 19:56:31 on dist-test-slave-gjg4"
I20240506 19:56:31.126546 12903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:31.146459 12903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:31.147313 12903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:31.147635 12903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:31.148700 12903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 19:56:31.150417 12903 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 19:56:31.150491 12903 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:31.150576 12903 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 19:56:31.150641 12903 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:31.168593 12763 catalog_manager.cc:1259] Loaded cluster ID: cab80095522b47ec9cab71d12c4cac98
I20240506 19:56:31.168802 12763 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: loading cluster ID for follower catalog manager: success
I20240506 19:56:31.174180 12763 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: acquiring CA information for follower catalog manager: success
I20240506 19:56:31.177772 12763 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 19:56:31.189883 12903 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.194:43527
I20240506 19:56:31.189946 13032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.194:43527 every 8 connection(s)
I20240506 19:56:31.191525 12903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/data/info.pb
I20240506 19:56:31.198763 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 12903
I20240506 19:56:31.198992 12483 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/instance
I20240506 19:56:31.204322 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.48.195:0
--local_ip_for_outbound_sockets=127.12.48.195
--webserver_interface=127.12.48.195
--webserver_port=0
--tserver_master_addrs=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240506 19:56:31.222543 13034 heartbeater.cc:347] Connected to a master server at 127.12.48.254:33495
I20240506 19:56:31.223102 13034 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:31.224375 13034 heartbeater.cc:511] Master 127.12.48.254:33495 requested a full tablet report, sending...
I20240506 19:56:31.231356 12521 ts_manager.cc:194] Registered new tserver with Master: 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:31.232674 12521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.48.194:59149
I20240506 19:56:31.233202 12894 heartbeater.cc:503] Master 127.12.48.254:33495 was elected leader, sending a full tablet report...
I20240506 19:56:31.234366 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:31.234530 13033 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:31.235174 13033 heartbeater.cc:511] Master 127.12.48.252:36867 requested a full tablet report, sending...
I20240506 19:56:31.235298 13035 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:31.235415 13035 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:31.235930 13035 heartbeater.cc:511] Master 127.12.48.253:36819 requested a full tablet report, sending...
I20240506 19:56:31.236429 12664 ts_manager.cc:194] Registered new tserver with Master: 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:31.237035 12592 ts_manager.cc:194] Registered new tserver with Master: 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:31.519105 13041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:31.519378 13041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:31.520349 13041 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:56:31.520982 12532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:31.521482 12601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
I20240506 19:56:31.522118 12650 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:56:31.522459 13041 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 19:56:31.621704 13040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:31.622052 13040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:31.622218 13040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:56:31.630239 13040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:31.630481 13040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.48.195
I20240506 19:56:31.640344 13040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.48.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.48.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.48.195
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:31.641899 13040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:31.643802 13040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 19:56:31.653030 13050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:31.653234 13050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:31.653949 13050 leader_election.cc:288] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:31.654616 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:31.654697 12532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:31.655493 12578 leader_election.cc:302] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27d4482cd59a4e5687992b1f8a177373; no voters: 190a2eabb30d4154960a44eb50a8a87d, 97f390a786034cddbff757779037f861
I20240506 19:56:31.656001 13050 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 19:56:31.656814 13047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:31.657424 13049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:31.658057 13046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:31.658756 13040 server_base.cc:1034] running on GCE node
I20240506 19:56:31.659484 13040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:31.660519 13040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:31.661744 13040 hybrid_clock.cc:648] HybridClock initialized: now 1715025391661678 us; error 40 us; skew 500 ppm
I20240506 19:56:31.662052 13040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:31.664364 13040 webserver.cc:460] Webserver started at http://127.12.48.195:41617/ using document root <none> and password file <none>
I20240506 19:56:31.665107 13040 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:31.665268 13040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:31.665586 13040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 19:56:31.668265 13040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data/instance:
uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
format_stamp: "Formatted at 2024-05-06 19:56:31 on dist-test-slave-gjg4"
I20240506 19:56:31.669035 13040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/instance:
uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
format_stamp: "Formatted at 2024-05-06 19:56:31 on dist-test-slave-gjg4"
I20240506 19:56:31.674650 13040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240506 19:56:31.678229 13056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:31.679770 13040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240506 19:56:31.679975 13040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal
uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
format_stamp: "Formatted at 2024-05-06 19:56:31 on dist-test-slave-gjg4"
I20240506 19:56:31.680174 13040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:31.698386 13040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:31.699221 13040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:31.699520 13040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:31.700508 13040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 19:56:31.702343 13040 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 19:56:31.702445 13040 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:31.702575 13040 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 19:56:31.702637 13040 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:31.734983 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.195:40535 every 8 connection(s)
I20240506 19:56:31.735011 13040 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.195:40535
I20240506 19:56:31.736765 13040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/data/info.pb
I20240506 19:56:31.739439 13034 heartbeater.cc:503] Master 127.12.48.254:33495 was elected leader, sending a full tablet report...
I20240506 19:56:31.746615 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 13040
I20240506 19:56:31.746819 12483 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/instance
I20240506 19:56:31.772373 13170 heartbeater.cc:347] Connected to a master server at 127.12.48.254:33495
I20240506 19:56:31.772739 13170 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:31.774166 13170 heartbeater.cc:511] Master 127.12.48.254:33495 requested a full tablet report, sending...
I20240506 19:56:31.777179 12522 ts_manager.cc:194] Registered new tserver with Master: 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:31.778128 12522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.48.195:39651
I20240506 19:56:31.778882 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:31.779036 13169 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:31.779538 13171 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:31.779608 13169 heartbeater.cc:511] Master 127.12.48.252:36867 requested a full tablet report, sending...
I20240506 19:56:31.779668 13171 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:31.780318 13171 heartbeater.cc:511] Master 127.12.48.253:36819 requested a full tablet report, sending...
I20240506 19:56:31.780745 12664 ts_manager.cc:194] Registered new tserver with Master: 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:31.781734 12592 ts_manager.cc:194] Registered new tserver with Master: 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:31.791834 12483 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 19:56:31.824218 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_590574fe06124541a13478de2bb622c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:31.824764 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2e9470e9692b4c5998aecb3fbcd1aca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:31.825515 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1d21319157c94657a00bda550a5941aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.826311 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_590574fe06124541a13478de2bb622c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.826370 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_90f59a210daf45b3b97fc602fc81a7a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.826817 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90f59a210daf45b3b97fc602fc81a7a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:31.827257 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:31.826310 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d21319157c94657a00bda550a5941aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.829353 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cc400f0eb6114db3aa8a4c75bfdd879f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:31.823982 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_90611788d80e4fb7a757d4295a5395d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.830037 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90611788d80e4fb7a757d4295a5395d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.830397 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4429e63f5fe9433c8e7bc3668c728223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.831056 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4429e63f5fe9433c8e7bc3668c728223 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:31.830037 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc400f0eb6114db3aa8a4c75bfdd879f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.842345 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_309671cba0304de8a6fa12b908704553"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.843154 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_309671cba0304de8a6fa12b908704553 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.845284 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_683b8e95e63840d397fd8bfcbccccc28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.846210 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_683b8e95e63840d397fd8bfcbccccc28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.848235 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c529110c9d554f42a66a73e307c33d9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:31.849074 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c529110c9d554f42a66a73e307c33d9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:31.913586 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_590574fe06124541a13478de2bb622c3" } new_table_name: "default.Table_62b8e57e79294ddcb95616004aa07b8c" modify_external_catalogs: true
I20240506 19:56:31.914433 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cc400f0eb6114db3aa8a4c75bfdd879f" } new_table_name: "default.Table_d4151e2391a3400e975dc074f56b98bc" modify_external_catalogs: true
I20240506 19:56:31.915750 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1d21319157c94657a00bda550a5941aa" } new_table_name: "default.Table_3d26f12533394b93b36d16ba5c3a78a5" modify_external_catalogs: true
I20240506 19:56:31.917001 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_90611788d80e4fb7a757d4295a5395d4" } new_table_name: "default.Table_b0dd8e4fa6b34b8c896d43613013c4e2" modify_external_catalogs: true
I20240506 19:56:31.917099 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4429e63f5fe9433c8e7bc3668c728223" } modify_external_catalogs: true
I20240506 19:56:31.917338 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4429e63f5fe9433c8e7bc3668c728223" } modify_external_catalogs: true
I20240506 19:56:31.918460 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c529110c9d554f42a66a73e307c33d9b" } modify_external_catalogs: true
I20240506 19:56:31.918676 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c529110c9d554f42a66a73e307c33d9b" } modify_external_catalogs: true
I20240506 19:56:31.919282 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_683b8e95e63840d397fd8bfcbccccc28" } modify_external_catalogs: true
I20240506 19:56:31.919463 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_683b8e95e63840d397fd8bfcbccccc28" } modify_external_catalogs: true
I20240506 19:56:31.924702 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2e9470e9692b4c5998aecb3fbcd1aca8" } modify_external_catalogs: true
I20240506 19:56:31.925027 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2e9470e9692b4c5998aecb3fbcd1aca8" } modify_external_catalogs: true
I20240506 19:56:31.926040 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_309671cba0304de8a6fa12b908704553" } modify_external_catalogs: true
I20240506 19:56:31.926322 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_309671cba0304de8a6fa12b908704553" } modify_external_catalogs: true
I20240506 19:56:31.928241 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_90f59a210daf45b3b97fc602fc81a7a7" } new_table_name: "default.Table_3ba870c181264335b5958783edd25dbb" modify_external_catalogs: true
W20240506 19:56:31.932888 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67645b4e51344266be626353efe80b71 has no consensus state
W20240506 19:56:31.932948 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86e61e20b54147c8b76f7991d7a5a7e7 has no consensus state
W20240506 19:56:31.933660 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79d60f4bf2d54180934d2db788c0beeb has no consensus state
I20240506 19:56:31.937126 12483 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu pid 12559
I20240506 19:56:31.938722 12623 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 19:56:31.942862 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5725b47e43244f0af881b07e145b3aa has no consensus state
W20240506 19:56:31.942914 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dda255d9661642dea4fe1fc7b98884f2 has no consensus state
I20240506 19:56:31.956900 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 133b872cbc704574b347136f74d06dfa
I20240506 19:56:31.973985 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 303f169d69f64f9ba93a98eb10981d0b
I20240506 19:56:31.974196 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1acfd9f6aa114ad6940184b9d100f400
I20240506 19:56:31.974962 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a8617db1a05846a9a427e5e1e841e6bb
I20240506 19:56:31.975896 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a009aa99ba834112836fc6c7b98f4907
W20240506 19:56:31.976840 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_62b8e57e79294ddcb95616004aa07b8c [id=ee08d3829f304e16b5b8994259bb11b0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 86e61e20b54147c8b76f7991d7a5a7e7 consensus state has no leader
I20240506 19:56:31.988740 12828 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b872cbc704574b347136f74d06dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:31.993588 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 (DEFAULT_TABLE table=default.Table_683b8e95e63840d397fd8bfcbccccc28 [id=ed415fa459fc400286eb6b68b20c663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.995397 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acfd9f6aa114ad6940184b9d100f400. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:31.990325 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b872cbc704574b347136f74d06dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:31.998255 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 133b872cbc704574b347136f74d06dfa (DEFAULT_TABLE table=default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 [id=988320f18e4b4b5c94d1222ab8682902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.993588 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (DEFAULT_TABLE table=default.Table_62b8e57e79294ddcb95616004aa07b8c [id=ee08d3829f304e16b5b8994259bb11b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.998523 12825 tablet_service.cc:1467] Processing CreateTablet for tablet a009aa99ba834112836fc6c7b98f4907 (DEFAULT_TABLE table=default.Table_309671cba0304de8a6fa12b908704553 [id=06c3b0ef665c41d1b46a44deb792ce4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.998833 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133b872cbc704574b347136f74d06dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:31.999079 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a009aa99ba834112836fc6c7b98f4907. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:31.999363 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e61e20b54147c8b76f7991d7a5a7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:31.994374 12964 tablet_service.cc:1467] Processing CreateTablet for tablet a009aa99ba834112836fc6c7b98f4907 (DEFAULT_TABLE table=default.Table_309671cba0304de8a6fa12b908704553 [id=06c3b0ef665c41d1b46a44deb792ce4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.995054 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 133b872cbc704574b347136f74d06dfa (DEFAULT_TABLE table=default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 [id=988320f18e4b4b5c94d1222ab8682902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.994712 12963 tablet_service.cc:1467] Processing CreateTablet for tablet a8617db1a05846a9a427e5e1e841e6bb (DEFAULT_TABLE table=default.Table_c529110c9d554f42a66a73e307c33d9b [id=a59147918fa8480dba5a27ff3cfe9c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.995841 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 67645b4e51344266be626353efe80b71 (DEFAULT_TABLE table=default.Table_3d26f12533394b93b36d16ba5c3a78a5 [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.995522 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 79d60f4bf2d54180934d2db788c0beeb (DEFAULT_TABLE table=default.Table_d4151e2391a3400e975dc074f56b98bc [id=d658e67fc24140119c44b42eee53ce36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.996165 12959 tablet_service.cc:1467] Processing CreateTablet for tablet dda255d9661642dea4fe1fc7b98884f2 (DEFAULT_TABLE table=default.Table_3ba870c181264335b5958783edd25dbb [id=2b7d2dfedf4d4169a2244e104dcd154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:31.994014 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 (DEFAULT_TABLE table=default.Table_683b8e95e63840d397fd8bfcbccccc28 [id=ed415fa459fc400286eb6b68b20c663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.001850 12958 tablet_service.cc:1467] Processing CreateTablet for tablet d5725b47e43244f0af881b07e145b3aa (DEFAULT_TABLE table=default.Table_b0dd8e4fa6b34b8c896d43613013c4e2 [id=a33185b46fbc424e9e81cef309eb2cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.003063 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a009aa99ba834112836fc6c7b98f4907. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.004523 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8617db1a05846a9a427e5e1e841e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.005540 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 303f169d69f64f9ba93a98eb10981d0b (DEFAULT_TABLE table=default.Table_4429e63f5fe9433c8e7bc3668c728223 [id=101526a5d12f4a908e7fcfedd3c8bd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.005813 13102 tablet_service.cc:1467] Processing CreateTablet for tablet a009aa99ba834112836fc6c7b98f4907 (DEFAULT_TABLE table=default.Table_309671cba0304de8a6fa12b908704553 [id=06c3b0ef665c41d1b46a44deb792ce4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.006722 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 133b872cbc704574b347136f74d06dfa (DEFAULT_TABLE table=default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 [id=988320f18e4b4b5c94d1222ab8682902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.007012 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 (DEFAULT_TABLE table=default.Table_683b8e95e63840d397fd8bfcbccccc28 [id=ed415fa459fc400286eb6b68b20c663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.007094 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.007321 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.007927 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.008662 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda255d9661642dea4fe1fc7b98884f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.004874 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 79d60f4bf2d54180934d2db788c0beeb (DEFAULT_TABLE table=default.Table_d4151e2391a3400e975dc074f56b98bc [id=d658e67fc24140119c44b42eee53ce36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.010489 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (DEFAULT_TABLE table=default.Table_62b8e57e79294ddcb95616004aa07b8c [id=ee08d3829f304e16b5b8994259bb11b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.011042 13101 tablet_service.cc:1467] Processing CreateTablet for tablet a8617db1a05846a9a427e5e1e841e6bb (DEFAULT_TABLE table=default.Table_c529110c9d554f42a66a73e307c33d9b [id=a59147918fa8480dba5a27ff3cfe9c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.011978 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acfd9f6aa114ad6940184b9d100f400. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.012861 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 79d60f4bf2d54180934d2db788c0beeb (DEFAULT_TABLE table=default.Table_d4151e2391a3400e975dc074f56b98bc [id=d658e67fc24140119c44b42eee53ce36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.013505 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d60f4bf2d54180934d2db788c0beeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.013576 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a009aa99ba834112836fc6c7b98f4907. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.010967 12824 tablet_service.cc:1467] Processing CreateTablet for tablet a8617db1a05846a9a427e5e1e841e6bb (DEFAULT_TABLE table=default.Table_c529110c9d554f42a66a73e307c33d9b [id=a59147918fa8480dba5a27ff3cfe9c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.016320 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.017414 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e61e20b54147c8b76f7991d7a5a7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.017587 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8617db1a05846a9a427e5e1e841e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.018316 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8617db1a05846a9a427e5e1e841e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.018399 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 303f169d69f64f9ba93a98eb10981d0b (DEFAULT_TABLE table=default.Table_4429e63f5fe9433c8e7bc3668c728223 [id=101526a5d12f4a908e7fcfedd3c8bd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.018935 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303f169d69f64f9ba93a98eb10981d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.019241 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133b872cbc704574b347136f74d06dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.020090 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d60f4bf2d54180934d2db788c0beeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.020192 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.020632 12820 tablet_service.cc:1467] Processing CreateTablet for tablet dda255d9661642dea4fe1fc7b98884f2 (DEFAULT_TABLE table=default.Table_3ba870c181264335b5958783edd25dbb [id=2b7d2dfedf4d4169a2244e104dcd154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.021207 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda255d9661642dea4fe1fc7b98884f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.021644 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 67645b4e51344266be626353efe80b71 (DEFAULT_TABLE table=default.Table_3d26f12533394b93b36d16ba5c3a78a5 [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.022150 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67645b4e51344266be626353efe80b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.022130 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 67645b4e51344266be626353efe80b71 (DEFAULT_TABLE table=default.Table_3d26f12533394b93b36d16ba5c3a78a5 [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.022603 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67645b4e51344266be626353efe80b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.022504 12819 tablet_service.cc:1467] Processing CreateTablet for tablet d5725b47e43244f0af881b07e145b3aa (DEFAULT_TABLE table=default.Table_b0dd8e4fa6b34b8c896d43613013c4e2 [id=a33185b46fbc424e9e81cef309eb2cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.023028 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5725b47e43244f0af881b07e145b3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.023607 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (DEFAULT_TABLE table=default.Table_62b8e57e79294ddcb95616004aa07b8c [id=ee08d3829f304e16b5b8994259bb11b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.023847 13097 tablet_service.cc:1467] Processing CreateTablet for tablet dda255d9661642dea4fe1fc7b98884f2 (DEFAULT_TABLE table=default.Table_3ba870c181264335b5958783edd25dbb [id=2b7d2dfedf4d4169a2244e104dcd154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.024152 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e61e20b54147c8b76f7991d7a5a7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.024345 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda255d9661642dea4fe1fc7b98884f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.024533 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.024770 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.024871 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67645b4e51344266be626353efe80b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.025264 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5725b47e43244f0af881b07e145b3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.026221 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.026717 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acfd9f6aa114ad6940184b9d100f400. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.027280 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133b872cbc704574b347136f74d06dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.027737 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303f169d69f64f9ba93a98eb10981d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.028537 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d60f4bf2d54180934d2db788c0beeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.092139 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_45c641026ff24500b8012bd465f985b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.092814 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45c641026ff24500b8012bd465f985b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.097402 13096 tablet_service.cc:1467] Processing CreateTablet for tablet d5725b47e43244f0af881b07e145b3aa (DEFAULT_TABLE table=default.Table_b0dd8e4fa6b34b8c896d43613013c4e2 [id=a33185b46fbc424e9e81cef309eb2cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.098070 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5725b47e43244f0af881b07e145b3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.100720 13244 tablet_bootstrap.cc:492] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.101001 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_99963fe4c795413c98b34872a5ac9b5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.105193 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99963fe4c795413c98b34872a5ac9b5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.108134 13244 tablet_bootstrap.cc:654] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.109802 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.109958 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.110033 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.110244 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.112324 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f9c09ec116c0499eace352aee64c8760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.112763 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2beea19b5bb94396b6941f206811de8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.112932 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9c09ec116c0499eace352aee64c8760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:32.113286 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2beea19b5bb94396b6941f206811de8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.114221 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b7d1593710c74a2ebec92bb6685fb378"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.114682 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7d1593710c74a2ebec92bb6685fb378 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.115319 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4cb98c91c897472c8bc3015a31c31871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.115783 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 303f169d69f64f9ba93a98eb10981d0b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.116299 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9439def23f6b4eaa8554197ebb5630a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.116560 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 133b872cbc704574b347136f74d06dfa from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:32.116755 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9439def23f6b4eaa8554197ebb5630a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.117017 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 303f169d69f64f9ba93a98eb10981d0b from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:32.115804 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cb98c91c897472c8bc3015a31c31871 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.118230 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_70c166a9bf7e4235a0edd6561255bb1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.118688 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70c166a9bf7e4235a0edd6561255bb1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.120080 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d36494da319a48e18299fbb1708e484b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.120539 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d36494da319a48e18299fbb1708e484b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.121527 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2a4c70f40666495ba552107168859093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.121958 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a4c70f40666495ba552107168859093 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:32.128276 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 133b872cbc704574b347136f74d06dfa because the tablet was not found. No further retry: Not found: Tablet not found: 133b872cbc704574b347136f74d06dfa
W20240506 19:56:32.131309 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 133b872cbc704574b347136f74d06dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133b872cbc704574b347136f74d06dfa already in progress: creating tablet
W20240506 19:56:32.131928 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a8617db1a05846a9a427e5e1e841e6bb because the tablet was not found. No further retry: Not found: Tablet not found: a8617db1a05846a9a427e5e1e841e6bb
W20240506 19:56:32.132721 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1acfd9f6aa114ad6940184b9d100f400 because the tablet was not found. No further retry: Not found: Tablet not found: 1acfd9f6aa114ad6940184b9d100f400
W20240506 19:56:32.133363 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 303f169d69f64f9ba93a98eb10981d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303f169d69f64f9ba93a98eb10981d0b already in progress: creating tablet
W20240506 19:56:32.134179 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1acfd9f6aa114ad6940184b9d100f400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1acfd9f6aa114ad6940184b9d100f400 already in progress: creating tablet
W20240506 19:56:32.134785 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a009aa99ba834112836fc6c7b98f4907 because the tablet was not found. No further retry: Not found: Tablet not found: a009aa99ba834112836fc6c7b98f4907
W20240506 19:56:32.136140 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 303f169d69f64f9ba93a98eb10981d0b because the tablet was not found. No further retry: Not found: Tablet not found: 303f169d69f64f9ba93a98eb10981d0b
W20240506 19:56:32.136821 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a8617db1a05846a9a427e5e1e841e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8617db1a05846a9a427e5e1e841e6bb already in progress: creating tablet
I20240506 19:56:32.109663 13244 log.cc:824] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Log is configured to *not* fsync() on all Append() calls
W20240506 19:56:32.142731 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a009aa99ba834112836fc6c7b98f4907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a009aa99ba834112836fc6c7b98f4907 already in progress: creating tablet
I20240506 19:56:32.146581 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet 67645b4e51344266be626353efe80b71 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.147954 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet dda255d9661642dea4fe1fc7b98884f2 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.150568 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3ba870c181264335b5958783edd25dbb" } new_table_name: "default.Table_9b9a6771ff0047b8be8fe070ec4f58c0" modify_external_catalogs: true
I20240506 19:56:32.152683 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b0dd8e4fa6b34b8c896d43613013c4e2" } new_table_name: "default.Table_287c7a0b9f224814b8567314ce935fac" modify_external_catalogs: true
I20240506 19:56:32.155607 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3d26f12533394b93b36d16ba5c3a78a5" } new_table_name: "default.Table_1ffe81a0bb2546408795d12a19f0c7e2" modify_external_catalogs: true
I20240506 19:56:32.156781 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62b8e57e79294ddcb95616004aa07b8c" } new_table_name: "default.Table_332cf026e46847e98be018e8b36dba81" modify_external_catalogs: true
W20240506 19:56:32.160156 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a8617db1a05846a9a427e5e1e841e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8617db1a05846a9a427e5e1e841e6bb already in progress: creating tablet
W20240506 19:56:32.160591 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 303f169d69f64f9ba93a98eb10981d0b because the tablet was not found. No further retry: Not found: Tablet not found: 303f169d69f64f9ba93a98eb10981d0b
W20240506 19:56:32.160853 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a009aa99ba834112836fc6c7b98f4907 because the tablet was not found. No further retry: Not found: Tablet not found: a009aa99ba834112836fc6c7b98f4907
W20240506 19:56:32.161080 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1acfd9f6aa114ad6940184b9d100f400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1acfd9f6aa114ad6940184b9d100f400 already in progress: creating tablet
I20240506 19:56:32.167755 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 303f169d69f64f9ba93a98eb10981d0b (DEFAULT_TABLE table=default.Table_4429e63f5fe9433c8e7bc3668c728223 [id=101526a5d12f4a908e7fcfedd3c8bd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.168367 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303f169d69f64f9ba93a98eb10981d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.171679 12483 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu with pid 12559
I20240506 19:56:32.176213 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b872cbc704574b347136f74d06dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.183957 13245 tablet_bootstrap.cc:492] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:32.184301 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 133b872cbc704574b347136f74d06dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133b872cbc704574b347136f74d06dfa already in progress: creating tablet
I20240506 19:56:32.187004 13249 tablet_bootstrap.cc:492] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.198045 13245 tablet_bootstrap.cc:654] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.199560 13245 log.cc:824] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:32.200966 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d4151e2391a3400e975dc074f56b98bc" } new_table_name: "default.Table_6b51fb3090cd44d991db17b13f049255" modify_external_catalogs: true
I20240506 19:56:32.201850 13244 tablet_bootstrap.cc:492] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.202244 13244 ts_tablet_manager.cc:1367] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.102s user 0.021s sys 0.038s
I20240506 19:56:32.202723 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_99963fe4c795413c98b34872a5ac9b5f" } modify_external_catalogs: true
I20240506 19:56:32.202932 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_99963fe4c795413c98b34872a5ac9b5f" } modify_external_catalogs: true
I20240506 19:56:32.203722 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2beea19b5bb94396b6941f206811de8c" } modify_external_catalogs: true
I20240506 19:56:32.203881 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2beea19b5bb94396b6941f206811de8c" } modify_external_catalogs: true
I20240506 19:56:32.204694 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f9c09ec116c0499eace352aee64c8760" } modify_external_catalogs: true
I20240506 19:56:32.204878 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f9c09ec116c0499eace352aee64c8760" } modify_external_catalogs: true
I20240506 19:56:32.205762 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 67645b4e51344266be626353efe80b71
I20240506 19:56:32.206319 12519 catalog_manager.cc:6554] ReplaceTablet: tablet 67645b4e51344266be626353efe80b71 deleted and replaced by tablet 6e60cddf90fe4891be3c60651ebbb184
I20240506 19:56:32.206666 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_45c641026ff24500b8012bd465f985b5" } modify_external_catalogs: true
I20240506 19:56:32.206816 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_45c641026ff24500b8012bd465f985b5" } modify_external_catalogs: true
I20240506 19:56:32.206862 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:32.207967 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 67645b4e51344266be626353efe80b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67645b4e51344266be626353efe80b71 already in progress: creating tablet
W20240506 19:56:32.208081 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dda255d9661642dea4fe1fc7b98884f2 consensus state has no leader
I20240506 19:56:32.208423 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b7d1593710c74a2ebec92bb6685fb378" } modify_external_catalogs: true
I20240506 19:56:32.208612 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b7d1593710c74a2ebec92bb6685fb378" } modify_external_catalogs: true
I20240506 19:56:32.209949 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet dda255d9661642dea4fe1fc7b98884f2
I20240506 19:56:32.210441 12518 catalog_manager.cc:6554] ReplaceTablet: tablet dda255d9661642dea4fe1fc7b98884f2 deleted and replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9
W20240506 19:56:32.210803 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e60cddf90fe4891be3c60651ebbb184 has no consensus state
W20240506 19:56:32.211187 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86e61e20b54147c8b76f7991d7a5a7e7 consensus state has no leader
I20240506 19:56:32.212057 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet dda255d9661642dea4fe1fc7b98884f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:32.212339 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5725b47e43244f0af881b07e145b3aa consensus state has no leader
I20240506 19:56:32.212414 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:32.212857 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet dda255d9661642dea4fe1fc7b98884f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda255d9661642dea4fe1fc7b98884f2 already in progress: creating tablet
W20240506 19:56:32.213402 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 67645b4e51344266be626353efe80b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67645b4e51344266be626353efe80b71 already in progress: creating tablet
I20240506 19:56:32.213490 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet dda255d9661642dea4fe1fc7b98884f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:32.214274 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet dda255d9661642dea4fe1fc7b98884f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda255d9661642dea4fe1fc7b98884f2 already in progress: creating tablet
I20240506 19:56:32.214738 12828 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.215739 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet dda255d9661642dea4fe1fc7b98884f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.216485 13249 tablet_bootstrap.cc:654] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:32.216476 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 67645b4e51344266be626353efe80b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67645b4e51344266be626353efe80b71 already in progress: creating tablet
W20240506 19:56:32.217195 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet dda255d9661642dea4fe1fc7b98884f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda255d9661642dea4fe1fc7b98884f2 already in progress: creating tablet
I20240506 19:56:32.218022 13249 log.cc:824] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:32.218021 13244 raft_consensus.cc:348] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.218582 13244 raft_consensus.cc:374] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.218703 13244 raft_consensus.cc:729] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.219743 13244 consensus_queue.cc:260] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.229387 13244 ts_tablet_manager.cc:1398] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.027s user 0.016s sys 0.006s
I20240506 19:56:32.232723 13244 tablet_bootstrap.cc:492] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.233256 12519 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f2ce5a3e3ca14df69d688c19465b1064
W20240506 19:56:32.233716 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79d60f4bf2d54180934d2db788c0beeb consensus state has no leader
I20240506 19:56:32.233808 12513 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b4e07fc6362b4baea07a8f332b571ed9
I20240506 19:56:32.234602 13244 tablet_bootstrap.cc:654] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.236125 13245 tablet_bootstrap.cc:492] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.236438 13245 ts_tablet_manager.cc:1367] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.053s user 0.017s sys 0.017s
I20240506 19:56:32.239714 13244 tablet_bootstrap.cc:492] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.239895 13244 ts_tablet_manager.cc:1367] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240506 19:56:32.240162 13249 tablet_bootstrap.cc:492] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.240541 13249 ts_tablet_manager.cc:1367] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.000s
I20240506 19:56:32.242420 13244 raft_consensus.cc:348] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.243309 13244 raft_consensus.cc:374] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.243578 13244 raft_consensus.cc:729] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.243948 13244 consensus_queue.cc:260] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.244593 13244 ts_tablet_manager.cc:1398] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:32.244871 13244 tablet_bootstrap.cc:492] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.247087 13244 tablet_bootstrap.cc:654] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.247367 13245 raft_consensus.cc:348] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.247839 13245 raft_consensus.cc:374] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.247959 13245 raft_consensus.cc:729] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.250000 13245 consensus_queue.cc:260] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.253973 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a (DEFAULT_TABLE table=default.Table_99963fe4c795413c98b34872a5ac9b5f [id=7a192bdbbc3b4571963f35dcd4b589d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.254563 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622008bbcf94ebcb2c2b78c7237b05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.257337 13245 ts_tablet_manager.cc:1398] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.021s user 0.010s sys 0.004s
I20240506 19:56:32.257444 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a (DEFAULT_TABLE table=default.Table_99963fe4c795413c98b34872a5ac9b5f [id=7a192bdbbc3b4571963f35dcd4b589d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.257771 13170 heartbeater.cc:503] Master 127.12.48.254:33495 was elected leader, sending a full tablet report...
I20240506 19:56:32.258034 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622008bbcf94ebcb2c2b78c7237b05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.258376 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a (DEFAULT_TABLE table=default.Table_99963fe4c795413c98b34872a5ac9b5f [id=7a192bdbbc3b4571963f35dcd4b589d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.259009 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622008bbcf94ebcb2c2b78c7237b05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.259080 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a (DEFAULT_TABLE table=default.Table_2beea19b5bb94396b6941f206811de8c [id=8c7219b61355477582e0d6d88b0ce18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.259581 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 75fd62bbf3494d06adb3d0493674558f (DEFAULT_TABLE table=default.Table_9439def23f6b4eaa8554197ebb5630a8 [id=a37fddd6fdbb47de9ac82b799503a40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.262115 12513 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_287c7a0b9f224814b8567314ce935fac [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 2 got 0
I20240506 19:56:32.263296 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 (DEFAULT_TABLE table=default.Table_f9c09ec116c0499eace352aee64c8760 [id=d399bfc7bfab4572a933a4a2838b2ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.263904 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0254b176e3c24a818ffdd384a4bbcf00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.265753 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.265731 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 75fd62bbf3494d06adb3d0493674558f (DEFAULT_TABLE table=default.Table_9439def23f6b4eaa8554197ebb5630a8 [id=a37fddd6fdbb47de9ac82b799503a40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.266212 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fd62bbf3494d06adb3d0493674558f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.269539 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fd62bbf3494d06adb3d0493674558f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.269737 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a (DEFAULT_TABLE table=default.Table_2beea19b5bb94396b6941f206811de8c [id=8c7219b61355477582e0d6d88b0ce18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.270238 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.273833 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 (DEFAULT_TABLE table=default.Table_f9c09ec116c0499eace352aee64c8760 [id=d399bfc7bfab4572a933a4a2838b2ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.274425 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0254b176e3c24a818ffdd384a4bbcf00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.279385 13249 raft_consensus.cc:348] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.279983 13249 raft_consensus.cc:374] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.280153 13249 raft_consensus.cc:729] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.281271 13249 consensus_queue.cc:260] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.283250 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 75fd62bbf3494d06adb3d0493674558f (DEFAULT_TABLE table=default.Table_9439def23f6b4eaa8554197ebb5630a8 [id=a37fddd6fdbb47de9ac82b799503a40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.283769 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fd62bbf3494d06adb3d0493674558f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.287317 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_287c7a0b9f224814b8567314ce935fac [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 2 got 0
I20240506 19:56:32.287590 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_332cf026e46847e98be018e8b36dba81 [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 2 got 0
I20240506 19:56:32.287739 13245 tablet_bootstrap.cc:492] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.287798 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:32.289680 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a (DEFAULT_TABLE table=default.Table_2beea19b5bb94396b6941f206811de8c [id=8c7219b61355477582e0d6d88b0ce18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.290124 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.290433 13249 ts_tablet_manager.cc:1398] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.050s user 0.008s sys 0.011s
I20240506 19:56:32.292536 13245 tablet_bootstrap.cc:654] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.293457 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet dda255d9661642dea4fe1fc7b98884f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.293871 13242 tablet_replica.cc:330] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:32.294400 13242 raft_consensus.cc:2229] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.294881 13242 raft_consensus.cc:2258] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.295325 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 3622008bbcf94ebcb2c2b78c7237b05a
I20240506 19:56:32.283253 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 (DEFAULT_TABLE table=default.Table_f9c09ec116c0499eace352aee64c8760 [id=d399bfc7bfab4572a933a4a2838b2ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.295856 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0254b176e3c24a818ffdd384a4bbcf00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.296862 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:32.297523 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 3622008bbcf94ebcb2c2b78c7237b05a because the tablet was not found. No further retry: Not found: Tablet not found: 3622008bbcf94ebcb2c2b78c7237b05a
I20240506 19:56:32.299655 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.300830 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.301534 13242 ts_tablet_manager.cc:1875] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:32.301532 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 3622008bbcf94ebcb2c2b78c7237b05a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3622008bbcf94ebcb2c2b78c7237b05a already in progress: creating tablet
I20240506 19:56:32.303303 13249 tablet_bootstrap.cc:492] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.304380 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 598998d27cc44e6a9c8cf4ccd5a2b00a
I20240506 19:56:32.305577 13249 tablet_bootstrap.cc:654] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.305822 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:32.306457 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 598998d27cc44e6a9c8cf4ccd5a2b00a already in progress: creating tablet
I20240506 19:56:32.307632 13245 tablet_bootstrap.cc:492] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.307750 13245 ts_tablet_manager.cc:1367] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.001s
I20240506 19:56:32.308773 13245 raft_consensus.cc:348] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.308938 13245 raft_consensus.cc:374] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.309015 13245 raft_consensus.cc:729] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.309257 13245 consensus_queue.cc:260] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.310245 13245 ts_tablet_manager.cc:1398] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:32.310534 13245 tablet_bootstrap.cc:492] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.310673 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.310978 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.312132 13245 tablet_bootstrap.cc:654] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.314033 13249 tablet_bootstrap.cc:492] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.314208 13249 ts_tablet_manager.cc:1367] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240506 19:56:32.314782 13242 ts_tablet_manager.cc:1888] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.314953 13242 log.cc:1196] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/dda255d9661642dea4fe1fc7b98884f2
I20240506 19:56:32.315277 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.315419 13242 ts_tablet_manager.cc:1909] T dda255d9661642dea4fe1fc7b98884f2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:32.315384 13249 raft_consensus.cc:348] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.315644 13249 raft_consensus.cc:374] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.315734 13243 tablet_replica.cc:330] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:32.315778 13249 raft_consensus.cc:729] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.316031 13249 consensus_queue.cc:260] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.316258 13243 raft_consensus.cc:2229] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:32.316854 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a because the tablet was not found. No further retry: Not found: Tablet not found: 598998d27cc44e6a9c8cf4ccd5a2b00a
I20240506 19:56:32.317502 13243 raft_consensus.cc:2258] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.319283 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.319538 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.319633 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet dda255d9661642dea4fe1fc7b98884f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.319824 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.319926 13249 ts_tablet_manager.cc:1398] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 19:56:32.320024 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b872cbc704574b347136f74d06dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.319538 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.320240 13244 tablet_bootstrap.cc:492] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.320430 13244 ts_tablet_manager.cc:1367] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.027s
I20240506 19:56:32.320468 13246 tablet_replica.cc:330] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:32.320979 13246 raft_consensus.cc:2229] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.321228 13244 raft_consensus.cc:348] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.321480 13244 raft_consensus.cc:374] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.321556 13244 raft_consensus.cc:729] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:32.321694 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 303f169d69f64f9ba93a98eb10981d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303f169d69f64f9ba93a98eb10981d0b already in progress: creating tablet
I20240506 19:56:32.321808 13244 consensus_queue.cc:260] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:32.321956 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a8617db1a05846a9a427e5e1e841e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8617db1a05846a9a427e5e1e841e6bb already in progress: creating tablet
I20240506 19:56:32.322324 13244 ts_tablet_manager.cc:1398] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:32.322649 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.324779 13246 raft_consensus.cc:2258] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.324847 13244 tablet_bootstrap.cc:492] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.326683 13244 tablet_bootstrap.cc:654] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.328889 13243 ts_tablet_manager.cc:1875] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.329181 13245 tablet_bootstrap.cc:492] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.329375 13245 ts_tablet_manager.cc:1367] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20240506 19:56:32.330202 13245 raft_consensus.cc:348] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.330427 13245 raft_consensus.cc:374] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.330507 13245 raft_consensus.cc:729] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.330667 13246 ts_tablet_manager.cc:1875] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.330832 13245 consensus_queue.cc:260] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.331363 13245 ts_tablet_manager.cc:1398] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:32.331673 13245 tablet_bootstrap.cc:492] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.332227 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 67645b4e51344266be626353efe80b71 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.332816 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet a8617db1a05846a9a427e5e1e841e6bb from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.333269 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 303f169d69f64f9ba93a98eb10981d0b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.333797 13245 tablet_bootstrap.cc:654] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.337483 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.339226 13245 tablet_bootstrap.cc:492] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.339406 13245 ts_tablet_manager.cc:1367] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240506 19:56:32.340306 13245 raft_consensus.cc:348] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.340512 13245 raft_consensus.cc:374] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.340601 13245 raft_consensus.cc:729] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.340869 13245 consensus_queue.cc:260] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.341289 13249 tablet_bootstrap.cc:492] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.341423 13245 ts_tablet_manager.cc:1398] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:32.341671 13245 tablet_bootstrap.cc:492] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.343209 13245 tablet_bootstrap.cc:654] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.343508 13249 tablet_bootstrap.cc:654] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:32.345978 12783 connection.cc:535] client connection to 127.12.48.253:36819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:56:32.346963 12895 heartbeater.cc:644] Failed to heartbeat to 127.12.48.253:36819 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:56:32.347383 12508 connection.cc:535] client connection to 127.12.48.253:36819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 19:56:32.349233 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet dda255d9661642dea4fe1fc7b98884f2 (table default.Table_9b9a6771ff0047b8be8fe070ec4f58c0 [id=2b7d2dfedf4d4169a2244e104dcd154d]) successfully deleted
W20240506 19:56:32.351547 12507 catalog_manager.cc:4797] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 3622008bbcf94ebcb2c2b78c7237b05a because the tablet was not found. No further retry: Not found: Tablet not found: 3622008bbcf94ebcb2c2b78c7237b05a
W20240506 19:56:32.351835 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 598998d27cc44e6a9c8cf4ccd5a2b00a already in progress: creating tablet
W20240506 19:56:32.356070 13254 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 19:56:32.241865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.48.253:36819 (local address 127.12.48.194:42183)
0506 19:56:32.276149 (+ 34284us) negotiation.cc:107] Waiting for socket to connect
0506 19:56:32.276171 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 19:56:32.276339 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 19:56:32.354507 (+ 78168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":34146,"thread_start_us":93582,"threads_started":1}
W20240506 19:56:32.356660 13035 heartbeater.cc:644] Failed to heartbeat to 127.12.48.253:36819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.48.253:36819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:56:32.360462 13257 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 19:56:32.280152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.48.253:36819 (local address 127.12.48.195:43429)
0506 19:56:32.299697 (+ 19545us) negotiation.cc:107] Waiting for socket to connect
0506 19:56:32.299719 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 19:56:32.299874 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 19:56:32.358774 (+ 58900us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19416,"thread_start_us":19831,"threads_started":1}
W20240506 19:56:32.361253 13171 heartbeater.cc:644] Failed to heartbeat to 127.12.48.253:36819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.48.253:36819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 19:56:32.362952 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_287c7a0b9f224814b8567314ce935fac [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 2 got 0
I20240506 19:56:32.369693 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_332cf026e46847e98be018e8b36dba81 [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 2 got 0
I20240506 19:56:32.370453 12963 tablet_service.cc:1467] Processing CreateTablet for tablet c51b52a98cec48fc959b01a2a0a8795c (DEFAULT_TABLE table=default.Table_2a4c70f40666495ba552107168859093 [id=513f5de1c20e4aa48bd9f216c869b5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.371153 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51b52a98cec48fc959b01a2a0a8795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.378332 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0254b176e3c24a818ffdd384a4bbcf00
I20240506 19:56:32.381347 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.384371 13102 tablet_service.cc:1467] Processing CreateTablet for tablet c51b52a98cec48fc959b01a2a0a8795c (DEFAULT_TABLE table=default.Table_2a4c70f40666495ba552107168859093 [id=513f5de1c20e4aa48bd9f216c869b5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.385109 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51b52a98cec48fc959b01a2a0a8795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.390074 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.391990 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d8c0cbd18a09476cb0895d1ce1c89de9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.392843 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.397598 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_eb361931fc51481590136c98bc041bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.398268 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb361931fc51481590136c98bc041bc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.398164 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_91b608707adc4be09b063e4928c32b38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.398247 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 162a5d3716c343139484a623105825a5 (DEFAULT_TABLE table=default.Table_d36494da319a48e18299fbb1708e484b [id=d1e6677412624f19917aae666e3e5184]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.398736 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162a5d3716c343139484a623105825a5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:32.398789 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91b608707adc4be09b063e4928c32b38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.399597 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4ef18fd530ba44f8956425df8676ebbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.400070 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ef18fd530ba44f8956425df8676ebbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.401144 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_06ec673eaab44720a193e5236fe7481d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.401615 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ec673eaab44720a193e5236fe7481d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.397518 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b4a1ca7c489b432f93a40d7be8d66723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.407100 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4a1ca7c489b432f93a40d7be8d66723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:32.414506 12508 consensus_peers.cc:485] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d -> Peer 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819): Couldn't send request to peer 27d4482cd59a4e5687992b1f8a177373. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:32.415453 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 81ea09efd4aa4019911dd23774f464a2 (DEFAULT_TABLE table=default.Table_70c166a9bf7e4235a0edd6561255bb1f [id=534590fb66294863bce3ed42ea46eb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.415974 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ea09efd4aa4019911dd23774f464a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.422248 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 (DEFAULT_TABLE table=default.Table_9b9a6771ff0047b8be8fe070ec4f58c0 [id=2b7d2dfedf4d4169a2244e104dcd154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.422699 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903ce1bc6ae7488d83bb0945424ed9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.432566 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 (DEFAULT_TABLE table=default.Table_1ffe81a0bb2546408795d12a19f0c7e2 [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.433238 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e60cddf90fe4891be3c60651ebbb184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.436316 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.436915 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 162a5d3716c343139484a623105825a5 (DEFAULT_TABLE table=default.Table_d36494da319a48e18299fbb1708e484b [id=d1e6677412624f19917aae666e3e5184]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.437434 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162a5d3716c343139484a623105825a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.439352 13246 ts_tablet_manager.cc:1888] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.439574 13246 log.cc:1196] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/dda255d9661642dea4fe1fc7b98884f2
I20240506 19:56:32.440287 12828 tablet_service.cc:1467] Processing CreateTablet for tablet c51b52a98cec48fc959b01a2a0a8795c (DEFAULT_TABLE table=default.Table_2a4c70f40666495ba552107168859093 [id=513f5de1c20e4aa48bd9f216c869b5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.440862 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51b52a98cec48fc959b01a2a0a8795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.441622 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 81ea09efd4aa4019911dd23774f464a2 (DEFAULT_TABLE table=default.Table_70c166a9bf7e4235a0edd6561255bb1f [id=534590fb66294863bce3ed42ea46eb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.442093 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ea09efd4aa4019911dd23774f464a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:32.442366 13267 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 19:56:32.347892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.48.253:36819 (local address 127.12.48.193:38241)
0506 19:56:32.434479 (+ 86587us) negotiation.cc:107] Waiting for socket to connect
0506 19:56:32.434783 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":86383,"thread_start_us":87535,"threads_started":1}
I20240506 19:56:32.442978 13246 ts_tablet_manager.cc:1909] T dda255d9661642dea4fe1fc7b98884f2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:32.447600 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a (DEFAULT_TABLE table=default.Table_4cb98c91c897472c8bc3015a31c31871 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.448035 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5513f4926e41a6b78f02a5126cc13a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:32.449663 12896 tablet.cc:2369] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:56:32.450560 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 (DEFAULT_TABLE table=default.Table_1ffe81a0bb2546408795d12a19f0c7e2 [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.451207 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e60cddf90fe4891be3c60651ebbb184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.452831 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 (DEFAULT_TABLE table=default.Table_9b9a6771ff0047b8be8fe070ec4f58c0 [id=2b7d2dfedf4d4169a2244e104dcd154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.453011 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a (DEFAULT_TABLE table=default.Table_4cb98c91c897472c8bc3015a31c31871 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:32.453435 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0254b176e3c24a818ffdd384a4bbcf00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0254b176e3c24a818ffdd384a4bbcf00 already in progress: creating tablet
I20240506 19:56:32.453608 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5513f4926e41a6b78f02a5126cc13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.454205 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_332cf026e46847e98be018e8b36dba81 [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 2 got 0
I20240506 19:56:32.456106 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903ce1bc6ae7488d83bb0945424ed9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.458850 13245 tablet_bootstrap.cc:492] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.459125 13245 ts_tablet_manager.cc:1367] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.117s user 0.033s sys 0.060s
I20240506 19:56:32.460029 13245 raft_consensus.cc:348] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:32.460361 13245 raft_consensus.cc:374] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.460484 13245 raft_consensus.cc:729] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.460835 13245 consensus_queue.cc:260] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:32.465276 13036 tablet.cc:2369] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:56:32.466297 12961 tablet_service.cc:1467] Processing CreateTablet for tablet b63857fa6004468cad193ed4f2a9d366 (DEFAULT_TABLE table=default.Table_91b608707adc4be09b063e4928c32b38 [id=8bd1845a4f4a437781f9b4535852aa5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.466743 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63857fa6004468cad193ed4f2a9d366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.467695 12959 tablet_service.cc:1467] Processing CreateTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 (DEFAULT_TABLE table=default.Table_eb361931fc51481590136c98bc041bc6 [id=031d0b422342478ba95396f085780f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.467958 13245 ts_tablet_manager.cc:1398] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:32.468094 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2abecd9ec7e4606b38c7d22477ddf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.469262 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a (DEFAULT_TABLE table=default.Table_4cb98c91c897472c8bc3015a31c31871 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.469763 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5513f4926e41a6b78f02a5126cc13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.470674 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 (DEFAULT_TABLE table=default.Table_9b9a6771ff0047b8be8fe070ec4f58c0 [id=2b7d2dfedf4d4169a2244e104dcd154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.471105 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903ce1bc6ae7488d83bb0945424ed9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.471824 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 10db9b8cad5045309dff3751cefce838 (DEFAULT_TABLE table=default.Table_06ec673eaab44720a193e5236fe7481d [id=5f25a2a3fe494e7f88d31b78f2e482c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.472237 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10db9b8cad5045309dff3751cefce838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.472684 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 10db9b8cad5045309dff3751cefce838 (DEFAULT_TABLE table=default.Table_06ec673eaab44720a193e5236fe7481d [id=5f25a2a3fe494e7f88d31b78f2e482c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.472816 12818 tablet_service.cc:1467] Processing CreateTablet for tablet b63857fa6004468cad193ed4f2a9d366 (DEFAULT_TABLE table=default.Table_91b608707adc4be09b063e4928c32b38 [id=8bd1845a4f4a437781f9b4535852aa5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.473104 12824 tablet_service.cc:1467] Processing CreateTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 (DEFAULT_TABLE table=default.Table_eb361931fc51481590136c98bc041bc6 [id=031d0b422342478ba95396f085780f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.473451 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 162a5d3716c343139484a623105825a5 (DEFAULT_TABLE table=default.Table_d36494da319a48e18299fbb1708e484b [id=d1e6677412624f19917aae666e3e5184]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.473507 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca44a4cb8344307b94c8fafba143836 (DEFAULT_TABLE table=default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.473830 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e (DEFAULT_TABLE table=default.Table_4ef18fd530ba44f8956425df8676ebbd [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.464375 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e (DEFAULT_TABLE table=default.Table_4ef18fd530ba44f8956425df8676ebbd [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.474167 12820 tablet_service.cc:1467] Processing CreateTablet for tablet ab4e6c606a634df58f2180361078d3d5 (DEFAULT_TABLE table=default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.474256 13244 tablet_bootstrap.cc:492] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.473104 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10db9b8cad5045309dff3751cefce838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.474990 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 (DEFAULT_TABLE table=default.Table_1ffe81a0bb2546408795d12a19f0c7e2 [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.475090 13244 ts_tablet_manager.cc:1367] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.150s user 0.037s sys 0.065s
I20240506 19:56:32.475373 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4e6c606a634df58f2180361078d3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.475337 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca44a4cb8344307b94c8fafba143836 (DEFAULT_TABLE table=default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.475515 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca44a4cb8344307b94c8fafba143836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.473249 12955 tablet_service.cc:1467] Processing CreateTablet for tablet ab4e6c606a634df58f2180361078d3d5 (DEFAULT_TABLE table=default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.476950 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63857fa6004468cad193ed4f2a9d366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.478502 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 81ea09efd4aa4019911dd23774f464a2 (DEFAULT_TABLE table=default.Table_70c166a9bf7e4235a0edd6561255bb1f [id=534590fb66294863bce3ed42ea46eb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.480278 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162a5d3716c343139484a623105825a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.481429 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdcdbb9cf3e4e9db6174bce7785565e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.483197 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2abecd9ec7e4606b38c7d22477ddf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.489199 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.490913 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e60cddf90fe4891be3c60651ebbb184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.493716 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4e6c606a634df58f2180361078d3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.494580 13244 raft_consensus.cc:348] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.494832 13244 raft_consensus.cc:374] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.494948 13244 raft_consensus.cc:729] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.495195 13244 consensus_queue.cc:260] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.496583 13244 ts_tablet_manager.cc:1398] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:32.496902 13244 tablet_bootstrap.cc:492] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.496933 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca44a4cb8344307b94c8fafba143836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.498553 13244 tablet_bootstrap.cc:654] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.501683 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ea09efd4aa4019911dd23774f464a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.506201 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdcdbb9cf3e4e9db6174bce7785565e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:32.514830 13178 tablet.cc:2369] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:56:32.516597 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet dda255d9661642dea4fe1fc7b98884f2 (table default.Table_9b9a6771ff0047b8be8fe070ec4f58c0 [id=2b7d2dfedf4d4169a2244e104dcd154d]) successfully deleted
I20240506 19:56:32.518097 13249 tablet_bootstrap.cc:492] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.521847 13245 tablet_bootstrap.cc:492] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.522760 13246 tablet_replica.cc:330] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:32.523366 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fcd8a4763edf472692c23bb235cb7852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.524219 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcd8a4763edf472692c23bb235cb7852 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.526722 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f158eaa0516d454b99196137dd7b16d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.527316 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f158eaa0516d454b99196137dd7b16d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.527220 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_71fac0ef50de48c096c6328f3f213053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.527694 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71fac0ef50de48c096c6328f3f213053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.528788 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet a8617db1a05846a9a427e5e1e841e6bb from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.528997 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_287c7a0b9f224814b8567314ce935fac" } new_table_name: "default.Table_f7e74cbd94724e53803a5429d2079986" modify_external_catalogs: true
I20240506 19:56:32.530594 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4cb98c91c897472c8bc3015a31c31871" } new_table_name: "default.Table_209a7bac6fc84e80afea650dd487e1e0" modify_external_catalogs: true
I20240506 19:56:32.530668 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9b9a6771ff0047b8be8fe070ec4f58c0" } new_table_name: "default.Table_675510c53e1848ed8176a861a1ace8c4" modify_external_catalogs: true
I20240506 19:56:32.530802 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.531494 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.532330 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_70c166a9bf7e4235a0edd6561255bb1f" } new_table_name: "default.Table_b7a9900250ea4635ab76bbb717bcb900" modify_external_catalogs: true
I20240506 19:56:32.533522 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet a8617db1a05846a9a427e5e1e841e6bb from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.534076 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1ffe81a0bb2546408795d12a19f0c7e2" } new_table_name: "default.Table_bfab2c8b01ba43aea99af3f2adba930e" modify_external_catalogs: true
W20240506 19:56:32.534838 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a009aa99ba834112836fc6c7b98f4907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a009aa99ba834112836fc6c7b98f4907 already in progress: creating tablet
I20240506 19:56:32.535389 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d36494da319a48e18299fbb1708e484b" } modify_external_catalogs: true
I20240506 19:56:32.535568 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d36494da319a48e18299fbb1708e484b" } modify_external_catalogs: true
I20240506 19:56:32.535892 13246 raft_consensus.cc:2229] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.536109 13246 raft_consensus.cc:2258] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.537904 13246 ts_tablet_manager.cc:1875] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.538494 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca44a4cb8344307b94c8fafba143836 (DEFAULT_TABLE table=default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.540567 13244 tablet_bootstrap.cc:492] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.540848 13244 ts_tablet_manager.cc:1367] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.027s
I20240506 19:56:32.541829 13244 raft_consensus.cc:348] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.542196 13244 raft_consensus.cc:374] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.542342 13244 raft_consensus.cc:729] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.542752 13244 consensus_queue.cc:260] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.543519 13244 ts_tablet_manager.cc:1398] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:32.543944 13244 tablet_bootstrap.cc:492] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.523646 13245 tablet_bootstrap.cc:654] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.545760 13244 tablet_bootstrap.cc:654] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.547691 13249 ts_tablet_manager.cc:1367] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.177s user 0.002s sys 0.006s
I20240506 19:56:32.548601 13249 raft_consensus.cc:348] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.549244 13249 raft_consensus.cc:374] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.549369 13249 raft_consensus.cc:729] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.549985 13249 consensus_queue.cc:260] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.552724 13249 ts_tablet_manager.cc:1398] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240506 19:56:32.555581 13249 tablet_bootstrap.cc:492] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.561794 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca44a4cb8344307b94c8fafba143836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.565544 13249 tablet_bootstrap.cc:654] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.565727 13095 tablet_service.cc:1467] Processing CreateTablet for tablet ab4e6c606a634df58f2180361078d3d5 (DEFAULT_TABLE table=default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.569646 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.570298 13242 tablet_replica.cc:330] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:32.570510 13242 raft_consensus.cc:2229] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.570685 13242 raft_consensus.cc:2258] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.571480 13242 ts_tablet_manager.cc:1875] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.571799 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9439def23f6b4eaa8554197ebb5630a8" } modify_external_catalogs: true
I20240506 19:56:32.572121 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9439def23f6b4eaa8554197ebb5630a8" } modify_external_catalogs: true
W20240506 19:56:32.572587 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5725b47e43244f0af881b07e145b3aa consensus state has no leader
I20240506 19:56:32.572979 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_06ec673eaab44720a193e5236fe7481d" } modify_external_catalogs: true
I20240506 19:56:32.573153 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_06ec673eaab44720a193e5236fe7481d" } modify_external_catalogs: true
W20240506 19:56:32.573504 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 903ce1bc6ae7488d83bb0945424ed9f9 consensus state has no leader
W20240506 19:56:32.573532 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d5513f4926e41a6b78f02a5126cc13a consensus state has no leader
I20240506 19:56:32.573829 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_eb361931fc51481590136c98bc041bc6" } modify_external_catalogs: true
I20240506 19:56:32.573990 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_eb361931fc51481590136c98bc041bc6" } modify_external_catalogs: true
I20240506 19:56:32.578115 13242 ts_tablet_manager.cc:1888] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.578289 13242 log.cc:1196] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/303f169d69f64f9ba93a98eb10981d0b
I20240506 19:56:32.578636 13242 ts_tablet_manager.cc:1909] T 303f169d69f64f9ba93a98eb10981d0b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:32.579084 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e (DEFAULT_TABLE table=default.Table_4ef18fd530ba44f8956425df8676ebbd [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.580483 13252 raft_consensus.cc:482] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.580749 13252 raft_consensus.cc:504] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.583045 13252 leader_election.cc:288] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:32.584434 13249 tablet_bootstrap.cc:492] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.584755 13249 ts_tablet_manager.cc:1367] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20240506 19:56:32.586365 13249 raft_consensus.cc:348] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.586802 13249 raft_consensus.cc:374] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.586994 13249 raft_consensus.cc:729] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.587512 13249 consensus_queue.cc:260] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.588608 13249 ts_tablet_manager.cc:1398] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:56:32.589183 13249 tablet_bootstrap.cc:492] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.591079 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4e6c606a634df58f2180361078d3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.591352 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 10db9b8cad5045309dff3751cefce838 (DEFAULT_TABLE table=default.Table_06ec673eaab44720a193e5236fe7481d [id=5f25a2a3fe494e7f88d31b78f2e482c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.592078 13249 tablet_bootstrap.cc:654] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.592931 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 303f169d69f64f9ba93a98eb10981d0b (table default.Table_4429e63f5fe9433c8e7bc3668c728223 [id=101526a5d12f4a908e7fcfedd3c8bd56]) successfully deleted
W20240506 19:56:32.595750 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81ea09efd4aa4019911dd23774f464a2 consensus state has no leader
W20240506 19:56:32.598209 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e60cddf90fe4891be3c60651ebbb184 consensus state has no leader
I20240506 19:56:32.598510 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 75fd62bbf3494d06adb3d0493674558f
I20240506 19:56:32.598652 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 162a5d3716c343139484a623105825a5
I20240506 19:56:32.599601 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fd62bbf3494d06adb3d0493674558f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.599898 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 162a5d3716c343139484a623105825a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.599978 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdcdbb9cf3e4e9db6174bce7785565e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.600524 13099 tablet_service.cc:1467] Processing CreateTablet for tablet b63857fa6004468cad193ed4f2a9d366 (DEFAULT_TABLE table=default.Table_91b608707adc4be09b063e4928c32b38 [id=8bd1845a4f4a437781f9b4535852aa5c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:32.600919 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 75fd62bbf3494d06adb3d0493674558f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75fd62bbf3494d06adb3d0493674558f already in progress: creating tablet
I20240506 19:56:32.601028 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10db9b8cad5045309dff3751cefce838. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:32.601198 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 162a5d3716c343139484a623105825a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 162a5d3716c343139484a623105825a5 already in progress: creating tablet
I20240506 19:56:32.601857 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fd62bbf3494d06adb3d0493674558f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.602169 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 162a5d3716c343139484a623105825a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.603078 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d2abecd9ec7e4606b38c7d22477ddf65
I20240506 19:56:32.604239 13103 tablet_service.cc:1467] Processing CreateTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 (DEFAULT_TABLE table=default.Table_eb361931fc51481590136c98bc041bc6 [id=031d0b422342478ba95396f085780f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.604413 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:32.606087 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d2abecd9ec7e4606b38c7d22477ddf65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2abecd9ec7e4606b38c7d22477ddf65 already in progress: creating tablet
I20240506 19:56:32.606154 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63857fa6004468cad193ed4f2a9d366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.606505 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.611466 13243 ts_tablet_manager.cc:1888] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.611713 13243 log.cc:1196] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/67645b4e51344266be626353efe80b71
I20240506 19:56:32.612993 13249 tablet_bootstrap.cc:492] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.613201 13249 ts_tablet_manager.cc:1367] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.004s
I20240506 19:56:32.614076 13249 raft_consensus.cc:348] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.614322 13249 raft_consensus.cc:374] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.614419 13249 raft_consensus.cc:729] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.614725 13249 consensus_queue.cc:260] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.617813 13249 ts_tablet_manager.cc:1398] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:56:32.619962 13258 raft_consensus.cc:482] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.620216 13258 raft_consensus.cc:504] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.622290 13258 leader_election.cc:288] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:32.624023 13249 tablet_bootstrap.cc:492] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.625770 13249 tablet_bootstrap.cc:654] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.627008 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fd62bbf3494d06adb3d0493674558f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.627415 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.627746 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 162a5d3716c343139484a623105825a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.629146 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet ab4e6c606a634df58f2180361078d3d5 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.629511 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 86e61e20b54147c8b76f7991d7a5a7e7 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.629766 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2abecd9ec7e4606b38c7d22477ddf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.631003 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e60cddf90fe4891be3c60651ebbb184 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.631515 13249 tablet_bootstrap.cc:492] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.631708 13249 ts_tablet_manager.cc:1367] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240506 19:56:32.632794 13249 raft_consensus.cc:348] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.633072 13249 raft_consensus.cc:374] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.633184 13249 raft_consensus.cc:729] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.633553 13249 consensus_queue.cc:260] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.635901 13243 ts_tablet_manager.cc:1909] T 67645b4e51344266be626353efe80b71 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:32.636032 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2a4c70f40666495ba552107168859093" } modify_external_catalogs: true
I20240506 19:56:32.636260 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2a4c70f40666495ba552107168859093" } modify_external_catalogs: true
I20240506 19:56:32.636921 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet dda255d9661642dea4fe1fc7b98884f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 903ce1bc6ae7488d83bb0945424ed9f9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.637696 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.638042 13242 tablet_replica.cc:330] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:32.638240 13242 raft_consensus.cc:2229] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.638460 13242 raft_consensus.cc:2258] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.638517 13249 ts_tablet_manager.cc:1398] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240506 19:56:32.639644 13242 ts_tablet_manager.cc:1875] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.642625 13249 tablet_bootstrap.cc:492] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.644491 13249 tablet_bootstrap.cc:654] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.644814 13246 ts_tablet_manager.cc:1888] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.651718 13233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:32.652042 13233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:32.656152 13242 ts_tablet_manager.cc:1888] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.656332 13242 log.cc:1196] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/67645b4e51344266be626353efe80b71
I20240506 19:56:32.656782 13242 ts_tablet_manager.cc:1909] T 67645b4e51344266be626353efe80b71 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:32.658416 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 18920e08b0de4f5292abb54e3432ee39 (DEFAULT_TABLE table=default.Table_fcd8a4763edf472692c23bb235cb7852 [id=229d13dce55e4738973b0180ddf23c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.658990 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18920e08b0de4f5292abb54e3432ee39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.662302 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 18920e08b0de4f5292abb54e3432ee39 (DEFAULT_TABLE table=default.Table_fcd8a4763edf472692c23bb235cb7852 [id=229d13dce55e4738973b0180ddf23c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.662837 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18920e08b0de4f5292abb54e3432ee39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.667474 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 67645b4e51344266be626353efe80b71 (table default.Table_bfab2c8b01ba43aea99af3f2adba930e [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:32.667913 13243 tablet_replica.cc:330] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:32.668548 13243 raft_consensus.cc:2229] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.669768 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 67645b4e51344266be626353efe80b71 (table default.Table_bfab2c8b01ba43aea99af3f2adba930e [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:32.671094 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5f69164a5a01441fa6dc72d34b878eb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.671026 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_76747eb355e74563ad356ac0b78eeb8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.671484 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_218a52f567cf4031bee1b7834387ac84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.671785 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f69164a5a01441fa6dc72d34b878eb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:32.671886 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76747eb355e74563ad356ac0b78eeb8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.671058 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4d2370baa11c44169a0260eac5ec2153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.674288 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d2370baa11c44169a0260eac5ec2153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:32.671893 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_218a52f567cf4031bee1b7834387ac84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.678913 13243 raft_consensus.cc:2258] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.679805 13243 ts_tablet_manager.cc:1875] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.687631 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_76561ee111374ba48c7245d01d1fe0ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.688342 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76561ee111374ba48c7245d01d1fe0ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.690011 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d60f4bf2d54180934d2db788c0beeb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:32.692010 12921 leader_election.cc:341] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:32.708532 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f553aaa2db8f41ae9ce0c146122e254d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.709264 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f553aaa2db8f41ae9ce0c146122e254d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.716329 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:32.716560 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_332cf026e46847e98be018e8b36dba81 [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 2 got 0
I20240506 19:56:32.721756 12957 tablet_service.cc:1467] Processing CreateTablet for tablet f017e635010040008be72b3813b5a4ed (DEFAULT_TABLE table=default.Table_f158eaa0516d454b99196137dd7b16d5 [id=4b9a2f4ea4b1460f89e8981e952e0d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.722051 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b872cbc704574b347136f74d06dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.721865 12962 tablet_service.cc:1467] Processing CreateTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b (DEFAULT_TABLE table=default.Table_71fac0ef50de48c096c6328f3f213053 [id=a60eb0d8de524b8aa111777bca63ca8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.722257 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f017e635010040008be72b3813b5a4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.722450 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.724532 12824 tablet_service.cc:1467] Processing CreateTablet for tablet f017e635010040008be72b3813b5a4ed (DEFAULT_TABLE table=default.Table_f158eaa0516d454b99196137dd7b16d5 [id=4b9a2f4ea4b1460f89e8981e952e0d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.725028 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f017e635010040008be72b3813b5a4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.726733 12822 tablet_service.cc:1467] Processing CreateTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b (DEFAULT_TABLE table=default.Table_71fac0ef50de48c096c6328f3f213053 [id=a60eb0d8de524b8aa111777bca63ca8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.727151 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.665841 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 18920e08b0de4f5292abb54e3432ee39 (DEFAULT_TABLE table=default.Table_fcd8a4763edf472692c23bb235cb7852 [id=229d13dce55e4738973b0180ddf23c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.728483 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18920e08b0de4f5292abb54e3432ee39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.729684 12821 tablet_service.cc:1467] Processing CreateTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 (DEFAULT_TABLE table=default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 [id=48be0d45bbf34e6187819fd3994a0afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.730095 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5adab0af6274ba0ad54402c5b1ff757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.677222 13245 tablet_bootstrap.cc:492] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.716300 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5725b47e43244f0af881b07e145b3aa" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:32.733362 13256 raft_consensus.cc:482] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.733990 13256 raft_consensus.cc:504] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.736119 13256 leader_election.cc:288] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:32.737917 13102 tablet_service.cc:1467] Processing CreateTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 (DEFAULT_TABLE table=default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 [id=48be0d45bbf34e6187819fd3994a0afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.738198 12955 tablet_service.cc:1467] Processing CreateTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 (DEFAULT_TABLE table=default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 [id=48be0d45bbf34e6187819fd3994a0afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.738462 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5adab0af6274ba0ad54402c5b1ff757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.738638 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5adab0af6274ba0ad54402c5b1ff757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.741765 13093 tablet_service.cc:1467] Processing CreateTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b (DEFAULT_TABLE table=default.Table_71fac0ef50de48c096c6328f3f213053 [id=a60eb0d8de524b8aa111777bca63ca8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.742267 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.742816 13124 raft_consensus.cc:2454] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:32.751648 13098 tablet_service.cc:1467] Processing CreateTablet for tablet f017e635010040008be72b3813b5a4ed (DEFAULT_TABLE table=default.Table_f158eaa0516d454b99196137dd7b16d5 [id=4b9a2f4ea4b1460f89e8981e952e0d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.752418 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f017e635010040008be72b3813b5a4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.758392 12783 leader_election.cc:302] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:32.758874 13252 raft_consensus.cc:2781] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:32.759038 13252 raft_consensus.cc:482] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:32.759151 13252 raft_consensus.cc:3037] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.762571 13252 raft_consensus.cc:504] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.763535 13252 leader_election.cc:288] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:32.764561 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5725b47e43244f0af881b07e145b3aa" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:32.764772 13124 raft_consensus.cc:3037] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.767943 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 7bdcdbb9cf3e4e9db6174bce7785565e from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.768325 13124 raft_consensus.cc:2454] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:56:32.768817 12783 leader_election.cc:302] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:32.769193 13252 raft_consensus.cc:2781] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:32.770184 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet b63857fa6004468cad193ed4f2a9d366 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:32.773480 13246 log.cc:1196] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/133b872cbc704574b347136f74d06dfa
I20240506 19:56:32.774170 13246 ts_tablet_manager.cc:1909] T 133b872cbc704574b347136f74d06dfa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:32.775848 13246 tablet_replica.cc:330] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:32.776216 13246 raft_consensus.cc:2229] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.776546 13246 raft_consensus.cc:2258] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.777699 13246 ts_tablet_manager.cc:1875] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.778880 13249 tablet_bootstrap.cc:492] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.779089 13249 ts_tablet_manager.cc:1367] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.032s
I20240506 19:56:32.780025 13249 raft_consensus.cc:348] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:32.780309 13249 raft_consensus.cc:374] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.780417 13249 raft_consensus.cc:729] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.780659 13249 consensus_queue.cc:260] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:32.782076 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5725b47e43244f0af881b07e145b3aa" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:32.783109 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5725b47e43244f0af881b07e145b3aa" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:32.785528 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 133b872cbc704574b347136f74d06dfa (table default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 [id=988320f18e4b4b5c94d1222ab8682902]) successfully deleted
W20240506 19:56:32.785483 12781 leader_election.cc:341] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:32.786304 12781 leader_election.cc:341] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:32.788441 13249 ts_tablet_manager.cc:1398] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:32.789033 13243 ts_tablet_manager.cc:1888] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.789216 13243 log.cc:1196] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a8617db1a05846a9a427e5e1e841e6bb
I20240506 19:56:32.789634 13243 ts_tablet_manager.cc:1909] T a8617db1a05846a9a427e5e1e841e6bb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:32.794286 13249 tablet_bootstrap.cc:492] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.795161 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5725b47e43244f0af881b07e145b3aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:32.777973 13245 ts_tablet_manager.cc:1367] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.256s user 0.002s sys 0.005s
I20240506 19:56:32.795691 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a8617db1a05846a9a427e5e1e841e6bb (table default.Table_c529110c9d554f42a66a73e307c33d9b [id=a59147918fa8480dba5a27ff3cfe9c1d]) successfully deleted
I20240506 19:56:32.793728 13256 raft_consensus.cc:482] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.796733 13252 raft_consensus.cc:686] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:56:32.797441 13252 consensus_queue.cc:237] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:32.798429 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0254b176e3c24a818ffdd384a4bbcf00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0254b176e3c24a818ffdd384a4bbcf00 already in progress: creating tablet
W20240506 19:56:32.798408 13057 leader_election.cc:341] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:32.797922 13256 raft_consensus.cc:504] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.799197 13256 leader_election.cc:288] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:32.800824 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 10db9b8cad5045309dff3751cefce838
W20240506 19:56:32.801090 13057 leader_election.cc:341] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a8617db1a05846a9a427e5e1e841e6bb
I20240506 19:56:32.802258 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 10db9b8cad5045309dff3751cefce838 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.802821 13249 tablet_bootstrap.cc:654] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.803097 13244 tablet_bootstrap.cc:492] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.803411 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 10db9b8cad5045309dff3751cefce838 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:32.803900 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 75fd62bbf3494d06adb3d0493674558f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75fd62bbf3494d06adb3d0493674558f already in progress: creating tablet
W20240506 19:56:32.804208 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 10db9b8cad5045309dff3751cefce838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10db9b8cad5045309dff3751cefce838 already in progress: creating tablet
W20240506 19:56:32.804543 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 162a5d3716c343139484a623105825a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 162a5d3716c343139484a623105825a5 already in progress: creating tablet
W20240506 19:56:32.805136 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d2abecd9ec7e4606b38c7d22477ddf65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2abecd9ec7e4606b38c7d22477ddf65 already in progress: creating tablet
I20240506 19:56:32.805148 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 6e60cddf90fe4891be3c60651ebbb184
I20240506 19:56:32.805919 13244 ts_tablet_manager.cc:1367] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.259s user 0.076s sys 0.114s
I20240506 19:56:32.806083 12517 catalog_manager.cc:6554] ReplaceTablet: tablet 6e60cddf90fe4891be3c60651ebbb184 deleted and replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246
I20240506 19:56:32.806891 13244 raft_consensus.cc:348] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.807099 13244 raft_consensus.cc:374] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.807196 13244 raft_consensus.cc:729] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.807442 13244 consensus_queue.cc:260] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.808039 13244 ts_tablet_manager.cc:1398] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:32.808305 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 with delete_type TABLET_DATA_DELETED (replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.808375 13244 tablet_bootstrap.cc:492] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.810181 13244 tablet_bootstrap.cc:654] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.810345 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 with delete_type TABLET_DATA_DELETED (replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.810292 13245 raft_consensus.cc:348] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.810528 13245 raft_consensus.cc:374] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.810634 13245 raft_consensus.cc:729] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.810703 13243 tablet_replica.cc:330] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:32.810889 13245 consensus_queue.cc:260] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.811002 13243 raft_consensus.cc:2229] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:32.811079 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 6e60cddf90fe4891be3c60651ebbb184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e60cddf90fe4891be3c60651ebbb184 already in progress: creating tablet
I20240506 19:56:32.811354 13243 raft_consensus.cc:2258] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.812288 13243 ts_tablet_manager.cc:1875] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.813374 13245 ts_tablet_manager.cc:1398] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240506 19:56:32.814544 13245 tablet_bootstrap.cc:492] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.815822 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 10db9b8cad5045309dff3751cefce838 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.816061 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 86e61e20b54147c8b76f7991d7a5a7e7
I20240506 19:56:32.816576 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 86e61e20b54147c8b76f7991d7a5a7e7 deleted and replaced by tablet 6156c6f582b747de973b357b6d071a46
I20240506 19:56:32.817055 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e61e20b54147c8b76f7991d7a5a7e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6156c6f582b747de973b357b6d071a46) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.817886 13258 raft_consensus.cc:482] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.818054 13258 raft_consensus.cc:504] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.818250 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e61e20b54147c8b76f7991d7a5a7e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6156c6f582b747de973b357b6d071a46) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.818521 13242 tablet_replica.cc:330] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:32.818730 13258 leader_election.cc:288] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:32.818729 13242 raft_consensus.cc:2229] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.818905 13242 raft_consensus.cc:2258] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.820900 13242 ts_tablet_manager.cc:1875] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.820971 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e61e20b54147c8b76f7991d7a5a7e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6156c6f582b747de973b357b6d071a46) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.823750 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133b872cbc704574b347136f74d06dfa" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:32.824527 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d60f4bf2d54180934d2db788c0beeb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:32.824788 13124 raft_consensus.cc:2454] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:32.825762 12922 leader_election.cc:302] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:32.826121 12921 leader_election.cc:341] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:32.826661 13258 raft_consensus.cc:2781] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:32.826814 13258 raft_consensus.cc:482] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:32.826954 13258 raft_consensus.cc:3037] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.827216 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f158eaa0516d454b99196137dd7b16d5" } new_table_name: "default.Table_726440f9935e4aaf893655b93fdeaf20" modify_external_catalogs: true
I20240506 19:56:32.828092 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8d2821d7c1594cbf9f72478d4c99a856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.828716 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d2821d7c1594cbf9f72478d4c99a856 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.828975 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.829278 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c51b52a98cec48fc959b01a2a0a8795c
I20240506 19:56:32.829536 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ab4e6c606a634df58f2180361078d3d5
I20240506 19:56:32.829735 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bfab2c8b01ba43aea99af3f2adba930e" } new_table_name: "default.Table_5c4383f23b354870b6fddac21d4a453a" modify_external_catalogs: true
I20240506 19:56:32.830030 12518 catalog_manager.cc:6554] ReplaceTablet: tablet ab4e6c606a634df58f2180361078d3d5 deleted and replaced by tablet d0e3260f46024386baac64cb8677883c
I20240506 19:56:32.830065 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_332cf026e46847e98be018e8b36dba81" } new_table_name: "default.Table_d90b50896a6b4231a2564273cefacca8" modify_external_catalogs: true
I20240506 19:56:32.830835 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_60cf0687fd504bdfb9f75986f69fd870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.831050 13258 raft_consensus.cc:504] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:32.831390 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60cf0687fd504bdfb9f75986f69fd870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.831410 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e6c606a634df58f2180361078d3d5 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e3260f46024386baac64cb8677883c) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.831651 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet c51b52a98cec48fc959b01a2a0a8795c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.831880 13258 leader_election.cc:288] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:32.832296 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet c51b52a98cec48fc959b01a2a0a8795c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.832332 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e6c606a634df58f2180361078d3d5 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e3260f46024386baac64cb8677883c) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.832453 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.832945 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.833717 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d60f4bf2d54180934d2db788c0beeb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:32.834039 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet c51b52a98cec48fc959b01a2a0a8795c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:32.834302 12921 leader_election.cc:341] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:32.834722 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9d0279aabb15419b8d8ff8dbc36f087f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.834830 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d60f4bf2d54180934d2db788c0beeb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:32.835081 13124 raft_consensus.cc:3037] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:32.835251 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d0279aabb15419b8d8ff8dbc36f087f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.835325 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e6c606a634df58f2180361078d3d5 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e3260f46024386baac64cb8677883c) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.837989 13124 raft_consensus.cc:2454] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:32.838395 12922 leader_election.cc:341] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 133b872cbc704574b347136f74d06dfa
I20240506 19:56:32.838608 12922 leader_election.cc:302] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:32.839331 12922 leader_election.cc:302] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:32.839730 13258 raft_consensus.cc:2726] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:32.839947 13258 raft_consensus.cc:2781] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:32.840148 13258 raft_consensus.cc:686] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:32.840673 13258 consensus_queue.cc:237] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.841537 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b63857fa6004468cad193ed4f2a9d366
I20240506 19:56:32.842198 12519 catalog_manager.cc:6554] ReplaceTablet: tablet b63857fa6004468cad193ed4f2a9d366 deleted and replaced by tablet 64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:32.842646 13242 ts_tablet_manager.cc:1888] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.842635 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_91b608707adc4be09b063e4928c32b38" } new_table_name: "default.Table_a84eed58b8c14359ad388d769fac448a" modify_external_catalogs: true
I20240506 19:56:32.842823 13242 log.cc:1196] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/86e61e20b54147c8b76f7991d7a5a7e7
I20240506 19:56:32.843003 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet b63857fa6004468cad193ed4f2a9d366 with delete_type TABLET_DATA_DELETED (replaced by tablet 64b7e42061cd487b8517447eb59ea8a1) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.843194 13242 ts_tablet_manager.cc:1909] T 86e61e20b54147c8b76f7991d7a5a7e7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:32.844167 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7bdcdbb9cf3e4e9db6174bce7785565e
I20240506 19:56:32.844779 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 7bdcdbb9cf3e4e9db6174bce7785565e deleted and replaced by tablet a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:32.845129 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_76561ee111374ba48c7245d01d1fe0ed" } new_table_name: "default.Table_dae4e46e818c4147960f5d606ce3b969" modify_external_catalogs: true
I20240506 19:56:32.845381 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e with delete_type TABLET_DATA_DELETED (replaced by tablet a16704567ef6454eaef886fb9cd599ce) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.846554 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_332cf026e46847e98be018e8b36dba81 [id=ee08d3829f304e16b5b8994259bb11b0]) successfully deleted
W20240506 19:56:32.846813 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c51b52a98cec48fc959b01a2a0a8795c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c51b52a98cec48fc959b01a2a0a8795c already in progress: creating tablet
I20240506 19:56:32.847003 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet b63857fa6004468cad193ed4f2a9d366 with delete_type TABLET_DATA_DELETED (replaced by tablet 64b7e42061cd487b8517447eb59ea8a1) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:32.847064 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ab4e6c606a634df58f2180361078d3d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab4e6c606a634df58f2180361078d3d5 already in progress: creating tablet
I20240506 19:56:32.847252 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e with delete_type TABLET_DATA_DELETED (replaced by tablet a16704567ef6454eaef886fb9cd599ce) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.847867 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet b63857fa6004468cad193ed4f2a9d366 with delete_type TABLET_DATA_DELETED (replaced by tablet 64b7e42061cd487b8517447eb59ea8a1) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.848093 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e with delete_type TABLET_DATA_DELETED (replaced by tablet a16704567ef6454eaef886fb9cd599ce) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.848474 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 with delete_type TABLET_DATA_DELETED (replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.849769 13245 tablet_bootstrap.cc:654] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:32.850246 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b63857fa6004468cad193ed4f2a9d366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b63857fa6004468cad193ed4f2a9d366 already in progress: creating tablet
W20240506 19:56:32.850960 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7bdcdbb9cf3e4e9db6174bce7785565e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bdcdbb9cf3e4e9db6174bce7785565e already in progress: creating tablet
W20240506 19:56:32.854141 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f017e635010040008be72b3813b5a4ed consensus state has no leader
I20240506 19:56:32.854523 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_76747eb355e74563ad356ac0b78eeb8e" } modify_external_catalogs: true
I20240506 19:56:32.854732 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_76747eb355e74563ad356ac0b78eeb8e" } modify_external_catalogs: true
W20240506 19:56:32.855352 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dc0ac37f67942fc8a31d8d43c839246 has no consensus state
I20240506 19:56:32.855700 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4d2370baa11c44169a0260eac5ec2153" } modify_external_catalogs: true
I20240506 19:56:32.855870 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4d2370baa11c44169a0260eac5ec2153" } modify_external_catalogs: true
I20240506 19:56:32.856747 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9" } modify_external_catalogs: true
I20240506 19:56:32.856945 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9" } modify_external_catalogs: true
W20240506 19:56:32.857187 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6156c6f582b747de973b357b6d071a46 has no consensus state
I20240506 19:56:32.863638 13245 tablet_bootstrap.cc:492] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.863838 13245 ts_tablet_manager.cc:1367] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240506 19:56:32.864620 13245 raft_consensus.cc:348] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.864871 13245 raft_consensus.cc:374] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.864946 13245 raft_consensus.cc:729] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:32.866730 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64b7e42061cd487b8517447eb59ea8a1 has no consensus state
W20240506 19:56:32.867228 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89eb4e2701894f32a85d2cd89da58ec2 has no consensus state
I20240506 19:56:32.869426 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:32.871456 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 (DEFAULT_TABLE table=default.Table_5f69164a5a01441fa6dc72d34b878eb2 [id=bc2b2ac5e9704dd48c4cd3620e7074e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.871896 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79486583025c42bfb1321f9cb6dd96f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.871829 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 (DEFAULT_TABLE table=default.Table_5f69164a5a01441fa6dc72d34b878eb2 [id=bc2b2ac5e9704dd48c4cd3620e7074e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.872228 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79486583025c42bfb1321f9cb6dd96f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.868111 13245 consensus_queue.cc:260] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.877400 13245 ts_tablet_manager.cc:1398] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240506 19:56:32.878104 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 (DEFAULT_TABLE table=default.Table_76747eb355e74563ad356ac0b78eeb8e [id=4ef402284c534d928551d10f8857c666]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.878544 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44de7153689b4dbd80dae87d3ee48a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.878839 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 (DEFAULT_TABLE table=default.Table_76747eb355e74563ad356ac0b78eeb8e [id=4ef402284c534d928551d10f8857c666]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.879273 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44de7153689b4dbd80dae87d3ee48a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.883576 13245 tablet_bootstrap.cc:492] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.885074 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5725b47e43244f0af881b07e145b3aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:32.886026 13245 tablet_bootstrap.cc:654] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.885545 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8617db1a05846a9a427e5e1e841e6bb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:32.886415 12847 raft_consensus.cc:2454] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:32.887109 13058 leader_election.cc:302] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:32.887661 13256 raft_consensus.cc:2726] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:32.888324 13244 tablet_bootstrap.cc:492] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:32.888341 13058 leader_election.cc:302] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745
I20240506 19:56:32.888506 13244 ts_tablet_manager.cc:1367] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.016s
I20240506 19:56:32.888767 13256 raft_consensus.cc:2781] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:32.888906 13256 raft_consensus.cc:482] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:32.889014 13256 raft_consensus.cc:3037] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.889523 13244 raft_consensus.cc:348] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.889737 13244 raft_consensus.cc:374] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.889797 13244 raft_consensus.cc:729] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.890017 13244 consensus_queue.cc:260] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.890508 13244 ts_tablet_manager.cc:1398] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:32.890918 13244 tablet_bootstrap.cc:492] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:32.892093 13256 raft_consensus.cc:504] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.892886 13244 tablet_bootstrap.cc:654] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.893824 13246 ts_tablet_manager.cc:1888] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.893988 13246 log.cc:1196] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1acfd9f6aa114ad6940184b9d100f400
W20240506 19:56:32.894383 13057 leader_election.cc:341] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a8617db1a05846a9a427e5e1e841e6bb
I20240506 19:56:32.894585 13246 ts_tablet_manager.cc:1909] T 1acfd9f6aa114ad6940184b9d100f400 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:32.895576 13256 leader_election.cc:288] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:32.895692 13246 tablet_replica.cc:330] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:32.895923 13246 raft_consensus.cc:2229] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.896108 13246 raft_consensus.cc:2258] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.896564 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8617db1a05846a9a427e5e1e841e6bb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:32.896649 13246 ts_tablet_manager.cc:1875] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.896761 12847 raft_consensus.cc:3037] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.897465 13283 raft_consensus.cc:482] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.897600 13283 raft_consensus.cc:504] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.898052 12513 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f6a6d62e91db49928743c8e118b2eda7
I20240506 19:56:32.898433 13283 leader_election.cc:288] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:32.899125 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d60f4bf2d54180934d2db788c0beeb" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:32.899945 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b5adab0af6274ba0ad54402c5b1ff757
I20240506 19:56:32.900557 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d60f4bf2d54180934d2db788c0beeb" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:32.900882 13124 raft_consensus.cc:2379] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef18e88b87064ed4b6c20f9325ebcc33 in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:56:32.901494 12783 leader_election.cc:302] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:32.901999 13283 raft_consensus.cc:3037] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.902967 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 (DEFAULT_TABLE table=default.Table_5f69164a5a01441fa6dc72d34b878eb2 [id=bc2b2ac5e9704dd48c4cd3620e7074e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.903441 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79486583025c42bfb1321f9cb6dd96f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.903753 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1acfd9f6aa114ad6940184b9d100f400 (table default.Table_683b8e95e63840d397fd8bfcbccccc28 [id=ed415fa459fc400286eb6b68b20c663c]) successfully deleted
W20240506 19:56:32.904299 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 67645b4e51344266be626353efe80b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67645b4e51344266be626353efe80b71 already in progress: creating tablet
I20240506 19:56:32.904866 13283 raft_consensus.cc:2726] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:32.905644 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 (DEFAULT_TABLE table=default.Table_76747eb355e74563ad356ac0b78eeb8e [id=4ef402284c534d928551d10f8857c666]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.906056 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44de7153689b4dbd80dae87d3ee48a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.906625 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.906757 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fdae8810fa204843b7e162d421c80856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.907341 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdae8810fa204843b7e162d421c80856 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.907828 12847 raft_consensus.cc:2454] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:32.908529 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:32.909201 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b5adab0af6274ba0ad54402c5b1ff757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5adab0af6274ba0ad54402c5b1ff757 already in progress: creating tablet
I20240506 19:56:32.909458 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.909605 13058 leader_election.cc:302] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745
W20240506 19:56:32.909840 13058 raft_consensus.cc:2670] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:32.910655 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f68de13c23f14d00b8f74290c11fcde3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.911233 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f68de13c23f14d00b8f74290c11fcde3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.913483 13243 ts_tablet_manager.cc:1888] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.913642 13243 log.cc:1196] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/dda255d9661642dea4fe1fc7b98884f2
I20240506 19:56:32.914021 13243 ts_tablet_manager.cc:1909] T dda255d9661642dea4fe1fc7b98884f2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:32.914855 13243 tablet_replica.cc:330] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:32.915066 13243 raft_consensus.cc:2229] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.915230 13243 raft_consensus.cc:2258] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.915549 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet dda255d9661642dea4fe1fc7b98884f2 (table default.Table_675510c53e1848ed8176a861a1ace8c4 [id=2b7d2dfedf4d4169a2244e104dcd154d]) successfully deleted
I20240506 19:56:32.916339 13249 tablet_bootstrap.cc:492] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.916472 13249 ts_tablet_manager.cc:1367] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.011s
I20240506 19:56:32.916780 13245 tablet_bootstrap.cc:492] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.916932 13245 ts_tablet_manager.cc:1367] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240506 19:56:32.917209 13249 raft_consensus.cc:348] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.917483 13249 raft_consensus.cc:374] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.917569 13249 raft_consensus.cc:729] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.917753 13245 raft_consensus.cc:348] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.917811 13249 consensus_queue.cc:260] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.917968 13245 raft_consensus.cc:374] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.918327 13249 ts_tablet_manager.cc:1398] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:32.918596 13249 tablet_bootstrap.cc:492] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.923002 13249 tablet_bootstrap.cc:654] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.923318 13245 raft_consensus.cc:729] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.923619 13245 consensus_queue.cc:260] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.925055 13245 ts_tablet_manager.cc:1398] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:32.926198 13245 tablet_bootstrap.cc:492] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.927979 13245 tablet_bootstrap.cc:654] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.933876 13243 ts_tablet_manager.cc:1875] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.935513 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:32.936460 12514 catalog_manager.cc:5484] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:32.937805 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:32.938308 13246 ts_tablet_manager.cc:1888] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.938486 13246 log.cc:1196] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a8617db1a05846a9a427e5e1e841e6bb
I20240506 19:56:32.938899 13246 ts_tablet_manager.cc:1909] T a8617db1a05846a9a427e5e1e841e6bb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:32.941649 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 67645b4e51344266be626353efe80b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e60cddf90fe4891be3c60651ebbb184) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.942395 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 44de7153689b4dbd80dae87d3ee48a83
I20240506 19:56:32.943107 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.943394 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.943528 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_08c048b0b7014bb1a6224dd0996369ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.944106 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08c048b0b7014bb1a6224dd0996369ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.946233 13245 tablet_bootstrap.cc:492] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:32.946410 13245 ts_tablet_manager.cc:1367] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240506 19:56:32.946398 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.947190 13245 raft_consensus.cc:348] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:32.947347 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 44de7153689b4dbd80dae87d3ee48a83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44de7153689b4dbd80dae87d3ee48a83 already in progress: creating tablet
I20240506 19:56:32.947429 13245 raft_consensus.cc:374] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.947501 13245 raft_consensus.cc:729] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.947773 13245 consensus_queue.cc:260] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.942405 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_69a8d24535d24338a12209fd418f1e1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:32.948863 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a8d24535d24338a12209fd418f1e1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.953778 13245 ts_tablet_manager.cc:1398] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:32.954293 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:32.954502 13249 tablet_bootstrap.cc:492] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.954697 13249 ts_tablet_manager.cc:1367] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.006s
I20240506 19:56:32.958011 13245 tablet_bootstrap.cc:492] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:32.959779 13245 tablet_bootstrap.cc:654] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.959460 12513 catalog_manager.cc:5484] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193). New cstate: current_term: 1 leader_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:32.960022 12513 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_f7e74cbd94724e53803a5429d2079986 [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 3 got 0
I20240506 19:56:32.961169 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 03e896b3ba7c4864a26cdfce76fadcfb (DEFAULT_TABLE table=default.Table_9d0279aabb15419b8d8ff8dbc36f087f [id=15e39c71182c4a089e9c950679153305]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.961663 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e896b3ba7c4864a26cdfce76fadcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.961768 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a8617db1a05846a9a427e5e1e841e6bb (table default.Table_c529110c9d554f42a66a73e307c33d9b [id=a59147918fa8480dba5a27ff3cfe9c1d]) successfully deleted
W20240506 19:56:32.962041 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0254b176e3c24a818ffdd384a4bbcf00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0254b176e3c24a818ffdd384a4bbcf00 already in progress: creating tablet
W20240506 19:56:32.962249 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 75fd62bbf3494d06adb3d0493674558f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75fd62bbf3494d06adb3d0493674558f already in progress: creating tablet
I20240506 19:56:32.962440 13256 raft_consensus.cc:482] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:32.962579 13256 raft_consensus.cc:504] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:32.962801 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d2abecd9ec7e4606b38c7d22477ddf65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2abecd9ec7e4606b38c7d22477ddf65 already in progress: creating tablet
W20240506 19:56:32.962988 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 162a5d3716c343139484a623105825a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 162a5d3716c343139484a623105825a5 already in progress: creating tablet
W20240506 19:56:32.963217 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 10db9b8cad5045309dff3751cefce838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10db9b8cad5045309dff3751cefce838 already in progress: creating tablet
I20240506 19:56:32.963253 13256 leader_election.cc:288] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:32.963961 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e61e20b54147c8b76f7991d7a5a7e7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:32.964249 12987 raft_consensus.cc:2454] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:32.964702 13057 leader_election.cc:302] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:32.964835 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 03e896b3ba7c4864a26cdfce76fadcfb (DEFAULT_TABLE table=default.Table_9d0279aabb15419b8d8ff8dbc36f087f [id=15e39c71182c4a089e9c950679153305]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:32.964911 13058 leader_election.cc:341] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Not found: Tablet not found: 86e61e20b54147c8b76f7991d7a5a7e7
I20240506 19:56:32.965034 13256 raft_consensus.cc:2781] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:32.965148 13256 raft_consensus.cc:482] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:32.965237 13256 raft_consensus.cc:3037] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.965251 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e896b3ba7c4864a26cdfce76fadcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.965396 13246 tablet_replica.cc:330] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:32.968089 13256 raft_consensus.cc:504] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.968395 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_f7e74cbd94724e53803a5429d2079986 [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 3 got 0
I20240506 19:56:32.969007 13246 raft_consensus.cc:2229] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.969635 13246 raft_consensus.cc:2258] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:32.969938 13058 leader_election.cc:341] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Not found: Tablet not found: 86e61e20b54147c8b76f7991d7a5a7e7
I20240506 19:56:32.970248 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 03e896b3ba7c4864a26cdfce76fadcfb (DEFAULT_TABLE table=default.Table_9d0279aabb15419b8d8ff8dbc36f087f [id=15e39c71182c4a089e9c950679153305]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.970628 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e896b3ba7c4864a26cdfce76fadcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.971628 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e61e20b54147c8b76f7991d7a5a7e7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:32.971863 12987 raft_consensus.cc:3037] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:32.971976 13249 raft_consensus.cc:348] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.972221 13249 raft_consensus.cc:374] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.972307 13249 raft_consensus.cc:729] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.972568 13249 consensus_queue.cc:260] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.973115 13249 ts_tablet_manager.cc:1398] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240506 19:56:32.973520 13249 tablet_bootstrap.cc:492] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.974994 12987 raft_consensus.cc:2454] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:32.975247 13249 tablet_bootstrap.cc:654] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:32.963965 13243 ts_tablet_manager.cc:1888] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.975764 13243 log.cc:1196] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/133b872cbc704574b347136f74d06dfa
I20240506 19:56:32.976140 13243 ts_tablet_manager.cc:1909] T 133b872cbc704574b347136f74d06dfa P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:32.976965 13057 leader_election.cc:302] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:32.977180 13057 raft_consensus.cc:2670] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:32.977557 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 133b872cbc704574b347136f74d06dfa (table default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 [id=988320f18e4b4b5c94d1222ab8682902]) successfully deleted
I20240506 19:56:32.977814 13243 tablet_replica.cc:330] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:32.978061 13243 raft_consensus.cc:2229] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:32.978104 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 10db9b8cad5045309dff3751cefce838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10db9b8cad5045309dff3751cefce838 already in progress: creating tablet
I20240506 19:56:32.978289 13243 raft_consensus.cc:2258] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:32.978379 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 6e60cddf90fe4891be3c60651ebbb184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e60cddf90fe4891be3c60651ebbb184 already in progress: creating tablet
I20240506 19:56:32.983211 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bbc623ea167c48929e77cee2b3d07aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:32.983412 13256 leader_election.cc:288] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:32.983880 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc623ea167c48929e77cee2b3d07aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:32.985930 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:32.987634 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet a8617db1a05846a9a427e5e1e841e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:32.987947 13242 tablet_replica.cc:330] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:32.988034 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet f017e635010040008be72b3813b5a4ed (table default.Table_726440f9935e4aaf893655b93fdeaf20 [id=4b9a2f4ea4b1460f89e8981e952e0d2d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f017e635010040008be72b3813b5a4ed consensus state has no leader
I20240506 19:56:32.988131 13242 raft_consensus.cc:2229] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:32.988343 13242 raft_consensus.cc:2258] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:32.989382 13242 ts_tablet_manager.cc:1875] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.982833 13249 tablet_bootstrap.cc:492] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:32.989799 13249 ts_tablet_manager.cc:1367] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240506 19:56:32.990128 13246 ts_tablet_manager.cc:1875] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:32.991881 13249 raft_consensus.cc:348] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.992169 13249 raft_consensus.cc:374] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:32.992297 13249 raft_consensus.cc:729] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:32.992621 13249 consensus_queue.cc:260] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:32.993911 13249 ts_tablet_manager.cc:1398] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:32.995775 13249 tablet_bootstrap.cc:492] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:32.996935 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e (DEFAULT_TABLE table=default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc [id=8b4499b016334533ab4366c2a4b1bd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:32.997087 13242 ts_tablet_manager.cc:1888] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:32.997218 13242 log.cc:1196] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a8617db1a05846a9a427e5e1e841e6bb
I20240506 19:56:32.997429 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.998577 13242 ts_tablet_manager.cc:1909] T a8617db1a05846a9a427e5e1e841e6bb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:33.003158 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:32.998113 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e (DEFAULT_TABLE table=default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc [id=8b4499b016334533ab4366c2a4b1bd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.003235 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a8617db1a05846a9a427e5e1e841e6bb (table default.Table_c529110c9d554f42a66a73e307c33d9b [id=a59147918fa8480dba5a27ff3cfe9c1d]) successfully deleted
I20240506 19:56:33.003677 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.004380 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 8596757ef75e433fb801f7679db5fefb (DEFAULT_TABLE table=default.Table_60cf0687fd504bdfb9f75986f69fd870 [id=a76c2b5f25a24c55a8379608a9857eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.004774 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8596757ef75e433fb801f7679db5fefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.005087 13249 tablet_bootstrap.cc:654] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.005511 13245 tablet_bootstrap.cc:492] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.005688 13245 ts_tablet_manager.cc:1367] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.002s
I20240506 19:56:33.006593 13245 raft_consensus.cc:348] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.006845 13245 raft_consensus.cc:374] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.006945 13245 raft_consensus.cc:729] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.007231 12959 tablet_service.cc:1467] Processing CreateTablet for tablet a16704567ef6454eaef886fb9cd599ce (DEFAULT_TABLE table=default.Table_4ef18fd530ba44f8956425df8676ebbd [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.007225 13245 consensus_queue.cc:260] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.007592 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16704567ef6454eaef886fb9cd599ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.009605 12821 tablet_service.cc:1467] Processing CreateTablet for tablet d0e3260f46024386baac64cb8677883c (DEFAULT_TABLE table=default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.010013 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e3260f46024386baac64cb8677883c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.013718 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 6156c6f582b747de973b357b6d071a46 (DEFAULT_TABLE table=default.Table_d90b50896a6b4231a2564273cefacca8 [id=ee08d3829f304e16b5b8994259bb11b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.014171 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6156c6f582b747de973b357b6d071a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.015112 13245 ts_tablet_manager.cc:1398] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:33.015476 13245 tablet_bootstrap.cc:492] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.017211 13245 tablet_bootstrap.cc:654] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.017650 12825 tablet_service.cc:1467] Processing CreateTablet for tablet a16704567ef6454eaef886fb9cd599ce (DEFAULT_TABLE table=default.Table_4ef18fd530ba44f8956425df8676ebbd [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.018117 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16704567ef6454eaef886fb9cd599ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.021678 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 8842523a97954e3f81ffd87530201d93 (DEFAULT_TABLE table=default.Table_8d2821d7c1594cbf9f72478d4c99a856 [id=b1277bb384294e7ea3cd5d0a52b5c813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.022039 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8842523a97954e3f81ffd87530201d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.027278 12819 tablet_service.cc:1467] Processing CreateTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 (DEFAULT_TABLE table=default.Table_218a52f567cf4031bee1b7834387ac84 [id=779ea3288b6d40ceb9636fdde3697472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.027709 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a223c33ba56c4ddbba46fedbce660ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:32.999222 13243 ts_tablet_manager.cc:1875] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.000974 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.001551 12960 tablet_service.cc:1467] Processing CreateTablet for tablet d0e3260f46024386baac64cb8677883c (DEFAULT_TABLE table=default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.029644 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc0ac37f67942fc8a31d8d43c839246 (DEFAULT_TABLE table=default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.029820 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e3260f46024386baac64cb8677883c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.030005 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc0ac37f67942fc8a31d8d43c839246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.001794 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 8596757ef75e433fb801f7679db5fefb (DEFAULT_TABLE table=default.Table_60cf0687fd504bdfb9f75986f69fd870 [id=a76c2b5f25a24c55a8379608a9857eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.002337 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 6156c6f582b747de973b357b6d071a46 (DEFAULT_TABLE table=default.Table_d90b50896a6b4231a2564273cefacca8 [id=ee08d3829f304e16b5b8994259bb11b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.033736 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6156c6f582b747de973b357b6d071a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.033640 12827 tablet_service.cc:1467] Processing CreateTablet for tablet e34e128329204418b7d6fb55762fb590 (DEFAULT_TABLE table=default.Table_f553aaa2db8f41ae9ce0c146122e254d [id=a147e46c173c402db02793b1e11de5b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.034019 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34e128329204418b7d6fb55762fb590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.037047 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 (DEFAULT_TABLE table=default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.037546 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b7e42061cd487b8517447eb59ea8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.037915 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8596757ef75e433fb801f7679db5fefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.041714 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 (DEFAULT_TABLE table=default.Table_dae4e46e818c4147960f5d606ce3b969 [id=098471374f5e4282b0a6a64b15b4f9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.042111 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89eb4e2701894f32a85d2cd89da58ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.051265 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 320e714ed6b64f2d84770e7182e4103a (DEFAULT_TABLE table=default.Table_f68de13c23f14d00b8f74290c11fcde3 [id=090dfa9462424be1b846e1cb05569156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.051976 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320e714ed6b64f2d84770e7182e4103a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.055378 12816 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 (DEFAULT_TABLE table=default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.056002 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe1c6ed89d643ecb4e97402d78167b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.059337 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 09856d2dd66043fbab08966d04e7b992 (DEFAULT_TABLE table=default.Table_69a8d24535d24338a12209fd418f1e1b [id=4021900d99ce4302b2de9b33a687e94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.060092 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09856d2dd66043fbab08966d04e7b992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.062965 12955 tablet_service.cc:1467] Processing CreateTablet for tablet e34e128329204418b7d6fb55762fb590 (DEFAULT_TABLE table=default.Table_f553aaa2db8f41ae9ce0c146122e254d [id=a147e46c173c402db02793b1e11de5b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.063504 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34e128329204418b7d6fb55762fb590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.063594 12956 tablet_service.cc:1467] Processing CreateTablet for tablet d843fe82af1148f4a9045ce8b09ad11d (DEFAULT_TABLE table=default.Table_bbc623ea167c48929e77cee2b3d07aaa [id=f6f5f08248a142d6929736e7872ae027]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.063925 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d843fe82af1148f4a9045ce8b09ad11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.069989 12961 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 (DEFAULT_TABLE table=default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.070454 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe1c6ed89d643ecb4e97402d78167b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.069993 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 320e714ed6b64f2d84770e7182e4103a (DEFAULT_TABLE table=default.Table_f68de13c23f14d00b8f74290c11fcde3 [id=090dfa9462424be1b846e1cb05569156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.074348 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (DEFAULT_TABLE table=default.Table_08c048b0b7014bb1a6224dd0996369ee [id=a8abbf675b16433da6b7139648a8fb26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.074857 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c18888ae3d6420ca1de3b13fc36cf6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.076610 13124 raft_consensus.cc:1261] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer ef18e88b87064ed4b6c20f9325ebcc33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:33.077617 13252 consensus_queue.cc:1035] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:33.086275 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320e714ed6b64f2d84770e7182e4103a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.088364 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 (DEFAULT_TABLE table=default.Table_dae4e46e818c4147960f5d606ce3b969 [id=098471374f5e4282b0a6a64b15b4f9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.088835 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89eb4e2701894f32a85d2cd89da58ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.090453 12987 raft_consensus.cc:3037] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.092423 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 (DEFAULT_TABLE table=default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.092828 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc0ac37f67942fc8a31d8d43c839246 (DEFAULT_TABLE table=default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.093183 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc0ac37f67942fc8a31d8d43c839246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.094437 12960 tablet_service.cc:1467] Processing CreateTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 (DEFAULT_TABLE table=default.Table_218a52f567cf4031bee1b7834387ac84 [id=779ea3288b6d40ceb9636fdde3697472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.103276 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b7e42061cd487b8517447eb59ea8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.104820 13246 ts_tablet_manager.cc:1888] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.104991 13246 log.cc:1196] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/86e61e20b54147c8b76f7991d7a5a7e7
I20240506 19:56:33.106104 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a223c33ba56c4ddbba46fedbce660ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.106175 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e (DEFAULT_TABLE table=default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc [id=8b4499b016334533ab4366c2a4b1bd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.106666 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.114154 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 8596757ef75e433fb801f7679db5fefb (DEFAULT_TABLE table=default.Table_60cf0687fd504bdfb9f75986f69fd870 [id=a76c2b5f25a24c55a8379608a9857eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.114671 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8596757ef75e433fb801f7679db5fefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.121749 12827 tablet_service.cc:1467] Processing CreateTablet for tablet d843fe82af1148f4a9045ce8b09ad11d (DEFAULT_TABLE table=default.Table_bbc623ea167c48929e77cee2b3d07aaa [id=f6f5f08248a142d6929736e7872ae027]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.122277 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d843fe82af1148f4a9045ce8b09ad11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.122206 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 8842523a97954e3f81ffd87530201d93 (DEFAULT_TABLE table=default.Table_8d2821d7c1594cbf9f72478d4c99a856 [id=b1277bb384294e7ea3cd5d0a52b5c813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.122903 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8842523a97954e3f81ffd87530201d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.125101 13256 raft_consensus.cc:482] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.125286 13256 raft_consensus.cc:504] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.126075 12987 raft_consensus.cc:1261] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Refusing update from remote peer ef18e88b87064ed4b6c20f9325ebcc33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:33.126133 13256 leader_election.cc:288] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:33.126489 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 6156c6f582b747de973b357b6d071a46 (DEFAULT_TABLE table=default.Table_d90b50896a6b4231a2564273cefacca8 [id=ee08d3829f304e16b5b8994259bb11b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.126960 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6156c6f582b747de973b357b6d071a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.126891 13252 consensus_queue.cc:1035] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:33.130177 13058 leader_election.cc:341] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Not found: Tablet not found: 67645b4e51344266be626353efe80b71
I20240506 19:56:33.131322 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (DEFAULT_TABLE table=default.Table_08c048b0b7014bb1a6224dd0996369ee [id=a8abbf675b16433da6b7139648a8fb26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.131801 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c18888ae3d6420ca1de3b13fc36cf6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:33.132138 13057 leader_election.cc:341] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 67645b4e51344266be626353efe80b71
I20240506 19:56:33.132272 13057 leader_election.cc:302] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.132719 13256 raft_consensus.cc:2726] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:33.138280 13095 tablet_service.cc:1467] Processing CreateTablet for tablet d0e3260f46024386baac64cb8677883c (DEFAULT_TABLE table=default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.139020 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e3260f46024386baac64cb8677883c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.143870 13246 ts_tablet_manager.cc:1909] T 86e61e20b54147c8b76f7991d7a5a7e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.147590 13246 tablet_replica.cc:330] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.147570 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 09856d2dd66043fbab08966d04e7b992 (DEFAULT_TABLE table=default.Table_69a8d24535d24338a12209fd418f1e1b [id=4021900d99ce4302b2de9b33a687e94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.147958 13246 raft_consensus.cc:2229] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.148351 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09856d2dd66043fbab08966d04e7b992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.148737 13093 tablet_service.cc:1467] Processing CreateTablet for tablet a16704567ef6454eaef886fb9cd599ce (DEFAULT_TABLE table=default.Table_4ef18fd530ba44f8956425df8676ebbd [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.149713 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16704567ef6454eaef886fb9cd599ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.154265 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 8842523a97954e3f81ffd87530201d93 (DEFAULT_TABLE table=default.Table_8d2821d7c1594cbf9f72478d4c99a856 [id=b1277bb384294e7ea3cd5d0a52b5c813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.154839 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8842523a97954e3f81ffd87530201d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.159369 13243 ts_tablet_manager.cc:1888] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.160138 13123 raft_consensus.cc:1261] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:33.161409 12847 raft_consensus.cc:1261] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:33.162453 13258 consensus_queue.cc:1035] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:33.166854 13243 log.cc:1196] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/86e61e20b54147c8b76f7991d7a5a7e7
I20240506 19:56:33.167312 13243 ts_tablet_manager.cc:1909] T 86e61e20b54147c8b76f7991d7a5a7e7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.167538 13287 consensus_queue.cc:1035] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:33.168272 13243 tablet_replica.cc:330] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.168504 13243 raft_consensus.cc:2229] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.168686 13243 raft_consensus.cc:2258] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.175462 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_d90b50896a6b4231a2564273cefacca8 [id=ee08d3829f304e16b5b8994259bb11b0]) successfully deleted
I20240506 19:56:33.175611 13243 ts_tablet_manager.cc:1875] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.179373 13103 tablet_service.cc:1467] Processing CreateTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 (DEFAULT_TABLE table=default.Table_218a52f567cf4031bee1b7834387ac84 [id=779ea3288b6d40ceb9636fdde3697472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.179936 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a223c33ba56c4ddbba46fedbce660ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.180733 13301 mvcc.cc:205] Tried to move back new op lower bound from 7024744010368471040 to 7024744009087365120. Current Snapshot: MvccSnapshot[applied={T|T < 7024744010368471040}]
I20240506 19:56:33.184578 13299 mvcc.cc:205] Tried to move back new op lower bound from 7024744010368471040 to 7024744009087365120. Current Snapshot: MvccSnapshot[applied={T|T < 7024744010368471040}]
I20240506 19:56:33.195037 13306 tablet.cc:1722] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:33.210253 13244 tablet_bootstrap.cc:492] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.210376 13303 mvcc.cc:205] Tried to move back new op lower bound from 7024744010368471040 to 7024744009087365120. Current Snapshot: MvccSnapshot[applied={T|T < 7024744010368471040}]
I20240506 19:56:33.210604 13244 ts_tablet_manager.cc:1367] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.320s user 0.004s sys 0.001s
I20240506 19:56:33.211747 13244 raft_consensus.cc:348] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.212076 13244 raft_consensus.cc:374] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.212206 13244 raft_consensus.cc:729] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.212572 13244 consensus_queue.cc:260] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.216048 13243 ts_tablet_manager.cc:1888] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.216250 13243 log.cc:1196] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1acfd9f6aa114ad6940184b9d100f400
I20240506 19:56:33.216725 13243 ts_tablet_manager.cc:1909] T 1acfd9f6aa114ad6940184b9d100f400 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.224045 13312 tablet.cc:1722] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:33.229355 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 320e714ed6b64f2d84770e7182e4103a (DEFAULT_TABLE table=default.Table_f68de13c23f14d00b8f74290c11fcde3 [id=090dfa9462424be1b846e1cb05569156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.228762 13309 tablet.cc:1722] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:33.230546 13094 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 (DEFAULT_TABLE table=default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.231648 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 09856d2dd66043fbab08966d04e7b992 (DEFAULT_TABLE table=default.Table_69a8d24535d24338a12209fd418f1e1b [id=4021900d99ce4302b2de9b33a687e94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.232946 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_f7e74cbd94724e53803a5429d2079986 [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 3 got 0
I20240506 19:56:33.233588 13243 tablet_replica.cc:330] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.233714 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09856d2dd66043fbab08966d04e7b992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.233789 13243 raft_consensus.cc:2229] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.234007 13243 raft_consensus.cc:2258] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.234802 13243 ts_tablet_manager.cc:1875] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.236212 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_f7e74cbd94724e53803a5429d2079986 [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 3 got 0
I20240506 19:56:33.236423 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:33.238600 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1acfd9f6aa114ad6940184b9d100f400 (table default.Table_683b8e95e63840d397fd8bfcbccccc28 [id=ed415fa459fc400286eb6b68b20c663c]) successfully deleted
W20240506 19:56:33.238971 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ab4e6c606a634df58f2180361078d3d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab4e6c606a634df58f2180361078d3d5 already in progress: creating tablet
W20240506 19:56:33.239188 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c51b52a98cec48fc959b01a2a0a8795c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c51b52a98cec48fc959b01a2a0a8795c already in progress: creating tablet
W20240506 19:56:33.239362 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b63857fa6004468cad193ed4f2a9d366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b63857fa6004468cad193ed4f2a9d366 already in progress: creating tablet
W20240506 19:56:33.239564 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7bdcdbb9cf3e4e9db6174bce7785565e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bdcdbb9cf3e4e9db6174bce7785565e already in progress: creating tablet
W20240506 19:56:33.239737 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b5adab0af6274ba0ad54402c5b1ff757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5adab0af6274ba0ad54402c5b1ff757 already in progress: creating tablet
W20240506 19:56:33.239967 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 44de7153689b4dbd80dae87d3ee48a83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44de7153689b4dbd80dae87d3ee48a83 already in progress: creating tablet
I20240506 19:56:33.241195 13308 tablet.cc:1722] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:33.216162 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc0ac37f67942fc8a31d8d43c839246 (DEFAULT_TABLE table=default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.243432 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc0ac37f67942fc8a31d8d43c839246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.217360 13102 tablet_service.cc:1467] Processing CreateTablet for tablet e34e128329204418b7d6fb55762fb590 (DEFAULT_TABLE table=default.Table_f553aaa2db8f41ae9ce0c146122e254d [id=a147e46c173c402db02793b1e11de5b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.245165 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34e128329204418b7d6fb55762fb590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.219415 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 (DEFAULT_TABLE table=default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.249862 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b7e42061cd487b8517447eb59ea8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.251471 13243 ts_tablet_manager.cc:1888] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.251605 13243 log.cc:1196] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a009aa99ba834112836fc6c7b98f4907
I20240506 19:56:33.251996 13243 ts_tablet_manager.cc:1909] T a009aa99ba834112836fc6c7b98f4907 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.252521 13243 tablet_replica.cc:330] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.252712 13243 raft_consensus.cc:2229] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.252869 13243 raft_consensus.cc:2258] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.226455 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 (DEFAULT_TABLE table=default.Table_dae4e46e818c4147960f5d606ce3b969 [id=098471374f5e4282b0a6a64b15b4f9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.227368 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 303f169d69f64f9ba93a98eb10981d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.228331 13101 tablet_service.cc:1467] Processing CreateTablet for tablet d843fe82af1148f4a9045ce8b09ad11d (DEFAULT_TABLE table=default.Table_bbc623ea167c48929e77cee2b3d07aaa [id=f6f5f08248a142d6929736e7872ae027]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.254539 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a009aa99ba834112836fc6c7b98f4907 (table default.Table_309671cba0304de8a6fa12b908704553 [id=06c3b0ef665c41d1b46a44deb792ce4f]) successfully deleted
I20240506 19:56:33.255146 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89eb4e2701894f32a85d2cd89da58ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.256060 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d843fe82af1148f4a9045ce8b09ad11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.258734 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320e714ed6b64f2d84770e7182e4103a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.258961 13243 ts_tablet_manager.cc:1875] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.262421 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe1c6ed89d643ecb4e97402d78167b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.266367 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.253:36819
--webserver_interface=127.12.48.253
--webserver_port=35887
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 19:56:33.207818 13310 tablet.cc:1722] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:33.276060 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2b51a8ac571347a896b8741066c330c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.276944 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b51a8ac571347a896b8741066c330c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.279517 13243 ts_tablet_manager.cc:1888] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.279707 13243 log.cc:1196] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/303f169d69f64f9ba93a98eb10981d0b
I20240506 19:56:33.280086 13243 ts_tablet_manager.cc:1909] T 303f169d69f64f9ba93a98eb10981d0b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.281234 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 303f169d69f64f9ba93a98eb10981d0b (table default.Table_4429e63f5fe9433c8e7bc3668c728223 [id=101526a5d12f4a908e7fcfedd3c8bd56]) successfully deleted
I20240506 19:56:33.292645 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (DEFAULT_TABLE table=default.Table_08c048b0b7014bb1a6224dd0996369ee [id=a8abbf675b16433da6b7139648a8fb26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.164305 13249 tablet_bootstrap.cc:492] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.294166 13249 ts_tablet_manager.cc:1367] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.298s user 0.018s sys 0.043s
I20240506 19:56:33.302985 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c18888ae3d6420ca1de3b13fc36cf6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.305871 12961 tablet_service.cc:1467] Processing CreateTablet for tablet a80834187f874661b611b43065eee943 (DEFAULT_TABLE table=default.Table_2b51a8ac571347a896b8741066c330c4 [id=c18e335476d443f9949f5f38fa4550ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.306365 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80834187f874661b611b43065eee943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.313508 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_218a52f567cf4031bee1b7834387ac84" } modify_external_catalogs: true
I20240506 19:56:33.313982 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_218a52f567cf4031bee1b7834387ac84" } modify_external_catalogs: true
I20240506 19:56:33.327983 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a223c33ba56c4ddbba46fedbce660ae2
I20240506 19:56:33.329792 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:33.330822 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a223c33ba56c4ddbba46fedbce660ae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a223c33ba56c4ddbba46fedbce660ae2 already in progress: creating tablet
I20240506 19:56:33.335911 13249 raft_consensus.cc:348] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.336465 13249 raft_consensus.cc:374] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.336576 13249 raft_consensus.cc:729] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.336922 13249 consensus_queue.cc:260] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.339864 13249 ts_tablet_manager.cc:1398] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240506 19:56:33.341801 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 598998d27cc44e6a9c8cf4ccd5a2b00a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.341995 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b51fb3090cd44d991db17b13f049255" } modify_external_catalogs: true
I20240506 19:56:33.342262 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b51fb3090cd44d991db17b13f049255" } modify_external_catalogs: true
I20240506 19:56:33.342868 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e60cddf90fe4891be3c60651ebbb184 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.343088 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.343355 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 0ca44a4cb8344307b94c8fafba143836 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.349551 13246 raft_consensus.cc:2258] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.350297 13244 ts_tablet_manager.cc:1398] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.140s user 0.001s sys 0.001s
I20240506 19:56:33.350569 13246 ts_tablet_manager.cc:1875] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:33.350431 12783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 127006us
I20240506 19:56:33.351563 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 10db9b8cad5045309dff3751cefce838 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.351758 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_60cf0687fd504bdfb9f75986f69fd870" } new_table_name: "default.Table_73b96b599455444fa8fead40db334a12" modify_external_catalogs: true
I20240506 19:56:33.351895 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.352612 12827 tablet_service.cc:1467] Processing CreateTablet for tablet a80834187f874661b611b43065eee943 (DEFAULT_TABLE table=default.Table_2b51a8ac571347a896b8741066c330c4 [id=c18e335476d443f9949f5f38fa4550ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.353163 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80834187f874661b611b43065eee943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.353399 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_08c048b0b7014bb1a6224dd0996369ee" } new_table_name: "default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8" modify_external_catalogs: true
I20240506 19:56:33.354867 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3d78abd7df2441f586f2f5aa221df837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.355443 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d78abd7df2441f586f2f5aa221df837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.356739 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bb95295253b14e0eaff2dceb4887634e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.357523 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4ef18fd530ba44f8956425df8676ebbd" } new_table_name: "default.Table_1af9d86b2ea64c83914a6ae32457b2e0" modify_external_catalogs: true
I20240506 19:56:33.358795 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e3807d65b604472889ea0fa7e4b5e69b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.359247 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3807d65b604472889ea0fa7e4b5e69b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.359740 13246 ts_tablet_manager.cc:1888] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.359881 13246 log.cc:1196] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/67645b4e51344266be626353efe80b71
I20240506 19:56:33.360252 13246 ts_tablet_manager.cc:1909] T 67645b4e51344266be626353efe80b71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:33.357249 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb95295253b14e0eaff2dceb4887634e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.362166 13249 tablet_bootstrap.cc:492] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.365013 13309 tablet.cc:1717] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Skipping requested alter to schema version 3, tablet already version 3
I20240506 19:56:33.366606 13246 tablet_replica.cc:330] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.366804 13246 raft_consensus.cc:2229] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.366999 13246 raft_consensus.cc:2258] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.367753 13246 ts_tablet_manager.cc:1875] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:33.368624 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a223c33ba56c4ddbba46fedbce660ae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a223c33ba56c4ddbba46fedbce660ae2 already in progress: creating tablet
I20240506 19:56:33.369891 13249 tablet_bootstrap.cc:654] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.370501 13310 tablet.cc:1717] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Skipping requested alter to schema version 3, tablet already version 3
I20240506 19:56:33.375973 13095 tablet_service.cc:1467] Processing CreateTablet for tablet a80834187f874661b611b43065eee943 (DEFAULT_TABLE table=default.Table_2b51a8ac571347a896b8741066c330c4 [id=c18e335476d443f9949f5f38fa4550ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.376442 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80834187f874661b611b43065eee943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.376724 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 86e61e20b54147c8b76f7991d7a5a7e7 (table default.Table_d90b50896a6b4231a2564273cefacca8 [id=ee08d3829f304e16b5b8994259bb11b0]) successfully deleted
W20240506 19:56:33.376997 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c51b52a98cec48fc959b01a2a0a8795c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c51b52a98cec48fc959b01a2a0a8795c already in progress: creating tablet
I20240506 19:56:33.377460 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:33.377259 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ab4e6c606a634df58f2180361078d3d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab4e6c606a634df58f2180361078d3d5 already in progress: creating tablet
W20240506 19:56:33.377825 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b63857fa6004468cad193ed4f2a9d366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b63857fa6004468cad193ed4f2a9d366 already in progress: creating tablet
W20240506 19:56:33.378058 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7bdcdbb9cf3e4e9db6174bce7785565e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bdcdbb9cf3e4e9db6174bce7785565e already in progress: creating tablet
W20240506 19:56:33.378250 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 6e60cddf90fe4891be3c60651ebbb184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e60cddf90fe4891be3c60651ebbb184 already in progress: creating tablet
W20240506 19:56:33.378448 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b5adab0af6274ba0ad54402c5b1ff757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5adab0af6274ba0ad54402c5b1ff757 already in progress: creating tablet
I20240506 19:56:33.379624 13245 tablet_bootstrap.cc:492] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.380072 13245 ts_tablet_manager.cc:1367] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.364s user 0.134s sys 0.151s
I20240506 19:56:33.380873 13245 raft_consensus.cc:348] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.381435 13245 raft_consensus.cc:374] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.381399 13306 tablet.cc:1722] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:33.381517 13245 raft_consensus.cc:729] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.382606 13245 consensus_queue.cc:260] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.384279 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0ca44a4cb8344307b94c8fafba143836
I20240506 19:56:33.384796 12513 catalog_manager.cc:6554] ReplaceTablet: tablet 0ca44a4cb8344307b94c8fafba143836 deleted and replaced by tablet 92951843c8bd446aa86436b65d9eb793
I20240506 19:56:33.385054 13245 ts_tablet_manager.cc:1398] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:33.385761 13245 tablet_bootstrap.cc:492] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.385797 12513 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]). Expected version 2 got 0
I20240506 19:56:33.388773 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca44a4cb8344307b94c8fafba143836 with delete_type TABLET_DATA_DELETED (replaced by tablet 92951843c8bd446aa86436b65d9eb793) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.390203 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca44a4cb8344307b94c8fafba143836 with delete_type TABLET_DATA_DELETED (replaced by tablet 92951843c8bd446aa86436b65d9eb793) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:33.390297 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0ca44a4cb8344307b94c8fafba143836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ca44a4cb8344307b94c8fafba143836 already in progress: creating tablet
I20240506 19:56:33.390470 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 67645b4e51344266be626353efe80b71 (table default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:33.350677 13244 tablet_bootstrap.cc:492] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:33.390746 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 44de7153689b4dbd80dae87d3ee48a83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44de7153689b4dbd80dae87d3ee48a83 already in progress: creating tablet
I20240506 19:56:33.391741 13246 ts_tablet_manager.cc:1888] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.391872 13246 log.cc:1196] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a009aa99ba834112836fc6c7b98f4907
I20240506 19:56:33.392223 13246 ts_tablet_manager.cc:1909] T a009aa99ba834112836fc6c7b98f4907 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.393165 13246 tablet_replica.cc:330] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.393432 13246 raft_consensus.cc:2229] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.393602 13246 raft_consensus.cc:2258] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.393793 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b
I20240506 19:56:33.394290 12514 catalog_manager.cc:6554] ReplaceTablet: tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b deleted and replaced by tablet 959f9e652bdd430898c1d3a95fe765f6
I20240506 19:56:33.394519 13245 tablet_bootstrap.cc:654] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:33.396411 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0ca44a4cb8344307b94c8fafba143836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ca44a4cb8344307b94c8fafba143836 already in progress: creating tablet
I20240506 19:56:33.395269 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36] alter complete (version 2)
I20240506 19:56:33.398296 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a009aa99ba834112836fc6c7b98f4907 (table default.Table_309671cba0304de8a6fa12b908704553 [id=06c3b0ef665c41d1b46a44deb792ce4f]) successfully deleted
I20240506 19:56:33.399783 13245 tablet_bootstrap.cc:492] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.399942 13245 ts_tablet_manager.cc:1367] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240506 19:56:33.400192 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b with delete_type TABLET_DATA_DELETED (replaced by tablet 959f9e652bdd430898c1d3a95fe765f6) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.400511 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b with delete_type TABLET_DATA_DELETED (replaced by tablet 959f9e652bdd430898c1d3a95fe765f6) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.400811 13245 raft_consensus.cc:348] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.401048 13245 raft_consensus.cc:374] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.401139 13245 raft_consensus.cc:729] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.401717 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca44a4cb8344307b94c8fafba143836 with delete_type TABLET_DATA_DELETED (replaced by tablet 92951843c8bd446aa86436b65d9eb793) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.401969 13245 consensus_queue.cc:260] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.402412 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b with delete_type TABLET_DATA_DELETED (replaced by tablet 959f9e652bdd430898c1d3a95fe765f6) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:33.402827 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b already in progress: creating tablet
W20240506 19:56:33.403596 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b already in progress: creating tablet
I20240506 19:56:33.405208 13245 ts_tablet_manager.cc:1398] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:33.405627 13245 tablet_bootstrap.cc:492] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.407552 13245 tablet_bootstrap.cc:654] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.407635 13244 tablet_bootstrap.cc:654] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.409039 13306 tablet.cc:1717] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Skipping requested alter to schema version 3, tablet already version 3
I20240506 19:56:33.412418 13246 ts_tablet_manager.cc:1875] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.415139 13244 tablet_bootstrap.cc:492] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.415333 13244 ts_tablet_manager.cc:1367] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.002s
W20240506 19:56:33.417861 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8596757ef75e433fb801f7679db5fefb consensus state has no leader
I20240506 19:56:33.418401 13249 tablet_bootstrap.cc:492] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
W20240506 19:56:33.418484 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c18888ae3d6420ca1de3b13fc36cf6c consensus state has no leader
I20240506 19:56:33.418545 13249 ts_tablet_manager.cc:1367] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240506 19:56:33.419298 13249 raft_consensus.cc:348] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.419564 13249 raft_consensus.cc:374] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.419661 13249 raft_consensus.cc:729] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.419931 13249 consensus_queue.cc:260] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.421245 13249 ts_tablet_manager.cc:1398] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:33.421571 13249 tablet_bootstrap.cc:492] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:33.423290 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a16704567ef6454eaef886fb9cd599ce consensus state has no leader
I20240506 19:56:33.424010 13249 tablet_bootstrap.cc:654] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.427120 13244 raft_consensus.cc:348] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.427476 13244 raft_consensus.cc:374] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.427549 13244 raft_consensus.cc:729] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.427436 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_726440f9935e4aaf893655b93fdeaf20" } new_table_name: "default.Table_c715baa1d25447d991935176e0f72c6b" modify_external_catalogs: true
I20240506 19:56:33.429769 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f553aaa2db8f41ae9ce0c146122e254d" } new_table_name: "default.Table_f44dcd3362d64342ad02f7d17b3f24d0" modify_external_catalogs: true
I20240506 19:56:33.430153 13244 consensus_queue.cc:260] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.430809 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d8c0cbd18a09476cb0895d1ce1c89de9" } modify_external_catalogs: true
I20240506 19:56:33.431449 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_f7e74cbd94724e53803a5429d2079986 [id=a33185b46fbc424e9e81cef309eb2cdb]). Expected version 3 got 0
I20240506 19:56:33.432049 13245 tablet_bootstrap.cc:492] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.432236 13245 ts_tablet_manager.cc:1367] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.008s
I20240506 19:56:33.432623 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a84eed58b8c14359ad388d769fac448a" } modify_external_catalogs: true
I20240506 19:56:33.432996 13245 raft_consensus.cc:348] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.433213 13245 raft_consensus.cc:374] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.433111 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dd14f50dc4aa41a297c6484b3d750ab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.433277 13245 raft_consensus.cc:729] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.433564 13245 consensus_queue.cc:260] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.434013 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e35ddd258fc64d49b50b2d6f2b1da1db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.434597 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd14f50dc4aa41a297c6484b3d750ab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:33.434595 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e35ddd258fc64d49b50b2d6f2b1da1db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.434964 13245 ts_tablet_manager.cc:1398] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:33.435372 13245 tablet_bootstrap.cc:492] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.435850 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a84eed58b8c14359ad388d769fac448a" } modify_external_catalogs: true
I20240506 19:56:33.436383 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d8c0cbd18a09476cb0895d1ce1c89de9" } modify_external_catalogs: true
I20240506 19:56:33.437047 13245 tablet_bootstrap.cc:654] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.440237 13246 ts_tablet_manager.cc:1888] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.440476 13246 log.cc:1196] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/303f169d69f64f9ba93a98eb10981d0b
I20240506 19:56:33.441018 13246 ts_tablet_manager.cc:1909] T 303f169d69f64f9ba93a98eb10981d0b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.445509 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 92951843c8bd446aa86436b65d9eb793 (DEFAULT_TABLE table=default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.445843 13244 ts_tablet_manager.cc:1398] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.030s user 0.001s sys 0.002s
I20240506 19:56:33.446097 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92951843c8bd446aa86436b65d9eb793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.448072 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 92951843c8bd446aa86436b65d9eb793 (DEFAULT_TABLE table=default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.448491 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92951843c8bd446aa86436b65d9eb793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.449355 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 92951843c8bd446aa86436b65d9eb793 (DEFAULT_TABLE table=default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.449744 13245 tablet_bootstrap.cc:492] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.449900 13245 ts_tablet_manager.cc:1367] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20240506 19:56:33.449951 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92951843c8bd446aa86436b65d9eb793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.450774 13245 raft_consensus.cc:348] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.450997 13245 raft_consensus.cc:374] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.451094 13245 raft_consensus.cc:729] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.451402 13245 consensus_queue.cc:260] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.453907 13256 raft_consensus.cc:482] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.454058 13256 raft_consensus.cc:504] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.454757 13256 leader_election.cc:288] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:33.455439 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fd62bbf3494d06adb3d0493674558f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:33.455523 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 303f169d69f64f9ba93a98eb10981d0b (table default.Table_4429e63f5fe9433c8e7bc3668c728223 [id=101526a5d12f4a908e7fcfedd3c8bd56]) successfully deleted
I20240506 19:56:33.455698 12988 raft_consensus.cc:2454] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:33.455839 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a223c33ba56c4ddbba46fedbce660ae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a223c33ba56c4ddbba46fedbce660ae2 already in progress: creating tablet
W20240506 19:56:33.456113 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b already in progress: creating tablet
I20240506 19:56:33.456167 13249 tablet_bootstrap.cc:492] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.456348 13249 ts_tablet_manager.cc:1367] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.002s
W20240506 19:56:33.456350 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0ca44a4cb8344307b94c8fafba143836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ca44a4cb8344307b94c8fafba143836 already in progress: creating tablet
I20240506 19:56:33.456338 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fd62bbf3494d06adb3d0493674558f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:33.457098 13057 leader_election.cc:302] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:33.457156 13249 raft_consensus.cc:348] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.457439 13249 raft_consensus.cc:374] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.457515 13249 raft_consensus.cc:729] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:33.457664 13058 leader_election.cc:341] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.457767 13249 consensus_queue.cc:260] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.458442 13249 ts_tablet_manager.cc:1398] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:33.458940 13245 ts_tablet_manager.cc:1398] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240506 19:56:33.459352 13249 tablet_bootstrap.cc:492] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.459563 13244 tablet_bootstrap.cc:492] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:33.461323 13244 tablet_bootstrap.cc:654] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.461454 13256 raft_consensus.cc:2781] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:33.461572 13256 raft_consensus.cc:482] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:33.461675 13256 raft_consensus.cc:3037] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.461961 13249 tablet_bootstrap.cc:654] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.463734 13245 tablet_bootstrap.cc:492] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.465453 13245 tablet_bootstrap.cc:654] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.464530 13256 raft_consensus.cc:504] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:33.477170 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f017e635010040008be72b3813b5a4ed consensus state has no leader
I20240506 19:56:33.477416 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_f7e74cbd94724e53803a5429d2079986 [id=a33185b46fbc424e9e81cef309eb2cdb] alter complete (version 3)
I20240506 19:56:33.477857 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8de44207890c4ceb8542f805a6f9b8da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.478044 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_71fac0ef50de48c096c6328f3f213053" } modify_external_catalogs: true
I20240506 19:56:33.478411 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_71fac0ef50de48c096c6328f3f213053" } modify_external_catalogs: true
W20240506 19:56:33.478785 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8de44207890c4ceb8542f805a6f9b8da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.480772 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b122543bad9446f9bd6b57a8336b17bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.481398 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b122543bad9446f9bd6b57a8336b17bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.484109 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 79d60f4bf2d54180934d2db788c0beeb
I20240506 19:56:33.485016 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d60f4bf2d54180934d2db788c0beeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.485003 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_247909120dec47d3a2d1729c04ed16a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.485345 13242 tablet_replica.cc:330] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:33.485553 13242 raft_consensus.cc:2229] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.485761 13242 raft_consensus.cc:2258] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:33.485729 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_247909120dec47d3a2d1729c04ed16a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.486994 13242 ts_tablet_manager.cc:1875] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.487583 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d60f4bf2d54180934d2db788c0beeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.487881 13243 tablet_replica.cc:330] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.488068 13243 raft_consensus.cc:2229] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:33.488710 13243 raft_consensus.cc:2258] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.489902 13243 ts_tablet_manager.cc:1875] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.490198 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 133b872cbc704574b347136f74d06dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.492053 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.494005 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fd62bbf3494d06adb3d0493674558f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.494881 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.496284 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.496820 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ddfa1d6244bd47dab51027f3945ed50f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.474169 13245 tablet_bootstrap.cc:492] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.497412 13245 ts_tablet_manager.cc:1367] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.005s
W20240506 19:56:33.497481 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddfa1d6244bd47dab51027f3945ed50f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.498328 13245 raft_consensus.cc:348] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.498633 13245 raft_consensus.cc:374] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.498682 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet a009aa99ba834112836fc6c7b98f4907 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.498723 13245 raft_consensus.cc:729] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.498975 13245 consensus_queue.cc:260] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.496388 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d60f4bf2d54180934d2db788c0beeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.499728 13246 tablet_replica.cc:330] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.499928 13246 raft_consensus.cc:2229] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.500221 13246 raft_consensus.cc:2258] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.500617 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fd62bbf3494d06adb3d0493674558f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:33.501315 13058 leader_election.cc:341] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.501557 13246 ts_tablet_manager.cc:1875] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.502664 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fd62bbf3494d06adb3d0493674558f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:33.502943 12988 raft_consensus.cc:3037] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.505007 13244 tablet_bootstrap.cc:492] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.505182 13244 ts_tablet_manager.cc:1367] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.013s
I20240506 19:56:33.506162 13244 raft_consensus.cc:348] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.506400 13244 raft_consensus.cc:374] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.506366 12988 raft_consensus.cc:2454] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:33.506491 13244 raft_consensus.cc:729] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.506788 13244 consensus_queue.cc:260] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.507022 13057 leader_election.cc:302] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.508531 13256 leader_election.cc:288] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:33.511022 13256 raft_consensus.cc:2781] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:33.511099 13243 ts_tablet_manager.cc:1888] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:33.511195 13244 ts_tablet_manager.cc:1398] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 19:56:33.512298 13244 tablet_bootstrap.cc:492] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:33.512724 13256 raft_consensus.cc:686] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:33.513334 13243 log.cc:1196] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/79d60f4bf2d54180934d2db788c0beeb
I20240506 19:56:33.513178 13256 consensus_queue.cc:237] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.514075 13243 ts_tablet_manager.cc:1909] T 79d60f4bf2d54180934d2db788c0beeb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.516386 13243 tablet_replica.cc:330] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.516649 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]) successfully deleted
I20240506 19:56:33.517581 13243 raft_consensus.cc:2229] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.517776 13243 raft_consensus.cc:2258] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.515437 13249 tablet_bootstrap.cc:492] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.518072 13249 ts_tablet_manager.cc:1367] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.010s
I20240506 19:56:33.508978 13246 ts_tablet_manager.cc:1888] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:33.518812 13246 log.cc:1196] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/79d60f4bf2d54180934d2db788c0beeb
I20240506 19:56:33.518916 13249 raft_consensus.cc:348] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.519143 13249 raft_consensus.cc:374] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.519243 13249 raft_consensus.cc:729] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.519677 13246 ts_tablet_manager.cc:1909] T 79d60f4bf2d54180934d2db788c0beeb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.520452 13242 ts_tablet_manager.cc:1888] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:33.520675 13243 ts_tablet_manager.cc:1875] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:33.521423 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e34e128329204418b7d6fb55762fb590 consensus state has no leader
I20240506 19:56:33.521435 13242 log.cc:1196] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/79d60f4bf2d54180934d2db788c0beeb
I20240506 19:56:33.521624 13252 raft_consensus.cc:482] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.521932 13242 ts_tablet_manager.cc:1909] T 79d60f4bf2d54180934d2db788c0beeb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:33.521811 13252 raft_consensus.cc:504] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.522969 13242 tablet_replica.cc:330] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:33.523157 13242 raft_consensus.cc:2229] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.523320 13242 raft_consensus.cc:2258] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.523576 13252 leader_election.cc:288] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:33.523890 13242 ts_tablet_manager.cc:1875] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:33.525892 12783 leader_election.cc:341] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a009aa99ba834112836fc6c7b98f4907
I20240506 19:56:33.525877 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]) successfully deleted
I20240506 19:56:33.526144 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 79d60f4bf2d54180934d2db788c0beeb (table default.Table_6b51fb3090cd44d991db17b13f049255 [id=d658e67fc24140119c44b42eee53ce36]) successfully deleted
I20240506 19:56:33.526105 13249 consensus_queue.cc:260] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:33.527416 12781 leader_election.cc:341] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a009aa99ba834112836fc6c7b98f4907
I20240506 19:56:33.527539 12781 leader_election.cc:302] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:33.527994 13252 raft_consensus.cc:2726] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:33.531673 13245 ts_tablet_manager.cc:1398] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.034s user 0.006s sys 0.002s
I20240506 19:56:33.532068 13245 tablet_bootstrap.cc:492] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.533838 13242 ts_tablet_manager.cc:1888] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.533954 13245 tablet_bootstrap.cc:654] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.534020 13242 log.cc:1196] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/133b872cbc704574b347136f74d06dfa
I20240506 19:56:33.534462 13242 ts_tablet_manager.cc:1909] T 133b872cbc704574b347136f74d06dfa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:33.534952 13249 ts_tablet_manager.cc:1398] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240506 19:56:33.535493 13242 tablet_replica.cc:330] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:33.535679 13249 tablet_bootstrap.cc:492] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.535693 13242 raft_consensus.cc:2229] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.535869 13242 raft_consensus.cc:2258] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.536336 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 133b872cbc704574b347136f74d06dfa (table default.Table_2e9470e9692b4c5998aecb3fbcd1aca8 [id=988320f18e4b4b5c94d1222ab8682902]) successfully deleted
I20240506 19:56:33.538508 13244 tablet_bootstrap.cc:654] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.538542 13249 tablet_bootstrap.cc:654] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.542838 13242 ts_tablet_manager.cc:1875] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.543087 13245 tablet_bootstrap.cc:492] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.543251 13245 ts_tablet_manager.cc:1367] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240506 19:56:33.544073 13245 raft_consensus.cc:348] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.544296 13245 raft_consensus.cc:374] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.544416 13245 raft_consensus.cc:729] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.544677 13245 consensus_queue.cc:260] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.545233 13245 ts_tablet_manager.cc:1398] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:33.545591 13245 tablet_bootstrap.cc:492] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.547060 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:33.548682 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.547298 13245 tablet_bootstrap.cc:654] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.548033 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.549504 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:33.550254 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 64b7e42061cd487b8517447eb59ea8a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64b7e42061cd487b8517447eb59ea8a1 already in progress: creating tablet
I20240506 19:56:33.550806 13243 ts_tablet_manager.cc:1888] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.551110 13243 log.cc:1196] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/598998d27cc44e6a9c8cf4ccd5a2b00a
I20240506 19:56:33.551599 13243 ts_tablet_manager.cc:1909] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.552044 13249 tablet_bootstrap.cc:492] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.552217 13249 ts_tablet_manager.cc:1367] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240506 19:56:33.553048 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 598998d27cc44e6a9c8cf4ccd5a2b00a (table default.Table_2beea19b5bb94396b6941f206811de8c [id=8c7219b61355477582e0d6d88b0ce18f]) successfully deleted
I20240506 19:56:33.553225 13249 raft_consensus.cc:348] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.553484 13249 raft_consensus.cc:374] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.553566 13249 raft_consensus.cc:729] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.553697 13243 tablet_replica.cc:330] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.553689 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 92951843c8bd446aa86436b65d9eb793
I20240506 19:56:33.553691 13242 ts_tablet_manager.cc:1888] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.553916 13242 log.cc:1196] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a009aa99ba834112836fc6c7b98f4907
I20240506 19:56:33.553934 13243 raft_consensus.cc:2229] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.554087 13243 raft_consensus.cc:2258] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.554344 13242 ts_tablet_manager.cc:1909] T a009aa99ba834112836fc6c7b98f4907 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:33.554920 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 92951843c8bd446aa86436b65d9eb793 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.555356 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 92951843c8bd446aa86436b65d9eb793 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.555409 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a009aa99ba834112836fc6c7b98f4907 (table default.Table_309671cba0304de8a6fa12b908704553 [id=06c3b0ef665c41d1b46a44deb792ce4f]) successfully deleted
W20240506 19:56:33.556012 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 64b7e42061cd487b8517447eb59ea8a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64b7e42061cd487b8517447eb59ea8a1 already in progress: creating tablet
W20240506 19:56:33.556208 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 92951843c8bd446aa86436b65d9eb793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92951843c8bd446aa86436b65d9eb793 already in progress: creating tablet
I20240506 19:56:33.556938 12517 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 959f9e652bdd430898c1d3a95fe765f6
W20240506 19:56:33.557060 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 92951843c8bd446aa86436b65d9eb793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92951843c8bd446aa86436b65d9eb793 already in progress: creating tablet
I20240506 19:56:33.553838 13249 consensus_queue.cc:260] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.554836 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 92951843c8bd446aa86436b65d9eb793 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.558292 13243 ts_tablet_manager.cc:1875] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.558861 13245 tablet_bootstrap.cc:492] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.559036 13245 ts_tablet_manager.cc:1367] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240506 19:56:33.560019 13245 raft_consensus.cc:348] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.560276 13245 raft_consensus.cc:374] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.560376 13245 raft_consensus.cc:729] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.560638 13245 consensus_queue.cc:260] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.561450 13245 ts_tablet_manager.cc:1398] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:33.562721 13245 tablet_bootstrap.cc:492] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.562963 13249 ts_tablet_manager.cc:1398] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240506 19:56:33.563477 13249 tablet_bootstrap.cc:492] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.565274 13249 tablet_bootstrap.cc:654] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.566643 13245 tablet_bootstrap.cc:654] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.571136 13243 ts_tablet_manager.cc:1888] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.573477 13244 tablet_bootstrap.cc:492] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.573732 13244 ts_tablet_manager.cc:1367] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.003s
I20240506 19:56:33.575403 13249 tablet_bootstrap.cc:492] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.575565 13249 ts_tablet_manager.cc:1367] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20240506 19:56:33.576229 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d91e484ef0cc4adb86af57d549a7e641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.576421 13249 raft_consensus.cc:348] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.576656 13249 raft_consensus.cc:374] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.576741 13249 raft_consensus.cc:729] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:33.576998 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d91e484ef0cc4adb86af57d549a7e641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.576997 13249 consensus_queue.cc:260] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.577587 13249 ts_tablet_manager.cc:1398] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240506 19:56:33.577548 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_73b96b599455444fa8fead40db334a12" } modify_external_catalogs: true
I20240506 19:56:33.577724 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_73b96b599455444fa8fead40db334a12" } modify_external_catalogs: true
I20240506 19:56:33.577972 13249 tablet_bootstrap.cc:492] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.579643 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.579895 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.580075 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.580257 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fd62bbf3494d06adb3d0493674558f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.580338 13243 log.cc:1196] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/75fd62bbf3494d06adb3d0493674558f
I20240506 19:56:33.580757 13243 ts_tablet_manager.cc:1909] T 75fd62bbf3494d06adb3d0493674558f P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.580840 13246 tablet_replica.cc:330] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.581049 13246 raft_consensus.cc:2229] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.581218 13246 raft_consensus.cc:2258] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.581806 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.582074 13249 tablet_bootstrap.cc:654] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.583153 13243 tablet_replica.cc:330] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.583338 13243 raft_consensus.cc:2229] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.583511 13243 raft_consensus.cc:2258] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.584393 13246 ts_tablet_manager.cc:1875] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.584487 13243 ts_tablet_manager.cc:1875] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.589484 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acfd9f6aa114ad6940184b9d100f400 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:31 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.589998 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 75fd62bbf3494d06adb3d0493674558f (table default.Table_9439def23f6b4eaa8554197ebb5630a8 [id=a37fddd6fdbb47de9ac82b799503a40d]) successfully deleted
I20240506 19:56:33.590222 13245 tablet_bootstrap.cc:492] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.590373 13245 ts_tablet_manager.cc:1367] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.003s
I20240506 19:56:33.591234 13245 raft_consensus.cc:348] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.591436 13243 ts_tablet_manager.cc:1888] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.591465 13245 raft_consensus.cc:374] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.591591 13245 raft_consensus.cc:729] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.591578 13243 log.cc:1196] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/3622008bbcf94ebcb2c2b78c7237b05a
I20240506 19:56:33.591847 13245 consensus_queue.cc:260] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.592020 13243 ts_tablet_manager.cc:1909] T 3622008bbcf94ebcb2c2b78c7237b05a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.592446 13245 ts_tablet_manager.cc:1398] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:33.593079 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 3622008bbcf94ebcb2c2b78c7237b05a (table default.Table_99963fe4c795413c98b34872a5ac9b5f [id=7a192bdbbc3b4571963f35dcd4b589d6]) successfully deleted
I20240506 19:56:33.595361 13244 raft_consensus.cc:348] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.595620 13244 raft_consensus.cc:374] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.595695 13244 raft_consensus.cc:729] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.595705 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 99dcc418075a4468a1d9a9887c007a20 (DEFAULT_TABLE table=default.Table_bb95295253b14e0eaff2dceb4887634e [id=0769cf069b8847faab1e5922412d544c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.596160 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99dcc418075a4468a1d9a9887c007a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.596473 13246 ts_tablet_manager.cc:1888] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.597092 13242 tablet_replica.cc:330] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:33.597404 13242 raft_consensus.cc:2229] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.597635 13242 raft_consensus.cc:2258] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.598237 13245 tablet_bootstrap.cc:492] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.599840 13244 consensus_queue.cc:260] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.599951 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 99dcc418075a4468a1d9a9887c007a20 (DEFAULT_TABLE table=default.Table_bb95295253b14e0eaff2dceb4887634e [id=0769cf069b8847faab1e5922412d544c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.600205 13246 log.cc:1196] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/598998d27cc44e6a9c8cf4ccd5a2b00a
I20240506 19:56:33.600608 13246 ts_tablet_manager.cc:1909] T 598998d27cc44e6a9c8cf4ccd5a2b00a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.600582 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 581aae0889654903933bef3bf1026954 (DEFAULT_TABLE table=default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.601107 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 (DEFAULT_TABLE table=default.Table_3d78abd7df2441f586f2f5aa221df837 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.602558 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 (DEFAULT_TABLE table=default.Table_3d78abd7df2441f586f2f5aa221df837 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.603088 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7f56fea34946e590dae0621b28f9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.604249 13244 ts_tablet_manager.cc:1398] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240506 19:56:33.605140 13246 tablet_replica.cc:330] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.605365 13246 raft_consensus.cc:2229] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.605567 13246 raft_consensus.cc:2258] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.606135 13245 tablet_bootstrap.cc:654] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.606575 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99dcc418075a4468a1d9a9887c007a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.607686 13246 ts_tablet_manager.cc:1875] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.608682 13242 ts_tablet_manager.cc:1875] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.610123 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 581aae0889654903933bef3bf1026954 (DEFAULT_TABLE table=default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.610576 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581aae0889654903933bef3bf1026954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.613847 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581aae0889654903933bef3bf1026954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.614609 13249 tablet_bootstrap.cc:492] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.614638 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7f56fea34946e590dae0621b28f9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.614799 13249 ts_tablet_manager.cc:1367] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.015s
I20240506 19:56:33.615689 13249 raft_consensus.cc:348] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.615919 13249 raft_consensus.cc:374] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.615999 13249 raft_consensus.cc:729] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.616315 13249 consensus_queue.cc:260] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.616847 13249 ts_tablet_manager.cc:1398] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:33.617194 13249 tablet_bootstrap.cc:492] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.618443 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 598998d27cc44e6a9c8cf4ccd5a2b00a (table default.Table_2beea19b5bb94396b6941f206811de8c [id=8c7219b61355477582e0d6d88b0ce18f]) successfully deleted
I20240506 19:56:33.618942 13249 tablet_bootstrap.cc:654] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.619504 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 (DEFAULT_TABLE table=default.Table_3d78abd7df2441f586f2f5aa221df837 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.620004 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7f56fea34946e590dae0621b28f9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.622840 13246 ts_tablet_manager.cc:1888] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.623009 13246 log.cc:1196] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/0254b176e3c24a818ffdd384a4bbcf00
I20240506 19:56:33.623396 13246 ts_tablet_manager.cc:1909] T 0254b176e3c24a818ffdd384a4bbcf00 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.624401 13246 tablet_replica.cc:330] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.624619 13246 raft_consensus.cc:2229] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.624776 13246 raft_consensus.cc:2258] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.624739 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 99dcc418075a4468a1d9a9887c007a20 (DEFAULT_TABLE table=default.Table_bb95295253b14e0eaff2dceb4887634e [id=0769cf069b8847faab1e5922412d544c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.625159 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99dcc418075a4468a1d9a9887c007a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.625942 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 581aae0889654903933bef3bf1026954 (DEFAULT_TABLE table=default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.626298 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581aae0889654903933bef3bf1026954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.633054 13246 ts_tablet_manager.cc:1875] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.635267 13244 tablet_bootstrap.cc:492] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:33.636881 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 0254b176e3c24a818ffdd384a4bbcf00 (table default.Table_f9c09ec116c0499eace352aee64c8760 [id=d399bfc7bfab4572a933a4a2838b2ab4]) successfully deleted
I20240506 19:56:33.637532 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b7a9900250ea4635ab76bbb717bcb900" } new_table_name: "default.Table_bedf0a0b6cd64f1bb0727feb78593f72" modify_external_catalogs: true
I20240506 19:56:33.638000 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8d2821d7c1594cbf9f72478d4c99a856" } new_table_name: "default.Table_0594be752cc441b1af65142e6d925488" modify_external_catalogs: true
I20240506 19:56:33.638849 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f1f139bb215b41778590c78acb73a3d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.639442 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1f139bb215b41778590c78acb73a3d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.640447 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8596757ef75e433fb801f7679db5fefb
I20240506 19:56:33.640686 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e35ddd258fc64d49b50b2d6f2b1da1db" } new_table_name: "default.Table_5bce626686754b56b9573ca9293b568f" modify_external_catalogs: true
I20240506 19:56:33.641675 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8de45342e9fc4eaa8efcb1c4d73e818a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.642210 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8de45342e9fc4eaa8efcb1c4d73e818a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.642554 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 8596757ef75e433fb801f7679db5fefb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.643100 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 8596757ef75e433fb801f7679db5fefb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.643373 13244 tablet_bootstrap.cc:654] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.643649 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet fbe1c6ed89d643ecb4e97402d78167b6 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.643692 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9d121b9e240b43cfbfea349bf45ef3a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.644213 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d121b9e240b43cfbfea349bf45ef3a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.646209 13242 ts_tablet_manager.cc:1888] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.646276 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet f017e635010040008be72b3813b5a4ed from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.646385 13242 log.cc:1196] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1acfd9f6aa114ad6940184b9d100f400
I20240506 19:56:33.646842 13242 ts_tablet_manager.cc:1909] T 1acfd9f6aa114ad6940184b9d100f400 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:33.647527 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1acfd9f6aa114ad6940184b9d100f400 (table default.Table_683b8e95e63840d397fd8bfcbccccc28 [id=ed415fa459fc400286eb6b68b20c663c]) successfully deleted
I20240506 19:56:33.647789 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 8596757ef75e433fb801f7679db5fefb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:33.647943 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8596757ef75e433fb801f7679db5fefb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8596757ef75e433fb801f7679db5fefb already in progress: creating tablet
W20240506 19:56:33.648169 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a009aa99ba834112836fc6c7b98f4907 because the tablet was not found. No further retry: Not found: Tablet not found: a009aa99ba834112836fc6c7b98f4907
W20240506 19:56:33.648893 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 64b7e42061cd487b8517447eb59ea8a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64b7e42061cd487b8517447eb59ea8a1 already in progress: creating tablet
W20240506 19:56:33.649623 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 92951843c8bd446aa86436b65d9eb793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92951843c8bd446aa86436b65d9eb793 already in progress: creating tablet
I20240506 19:56:33.649992 13243 tablet_replica.cc:330] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.650242 13243 raft_consensus.cc:2229] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.650478 13243 raft_consensus.cc:2258] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.651234 13243 ts_tablet_manager.cc:1875] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.655550 13244 tablet_bootstrap.cc:492] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.655722 13244 ts_tablet_manager.cc:1367] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.001s
I20240506 19:56:33.656606 13244 raft_consensus.cc:348] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.656859 13244 raft_consensus.cc:374] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.656961 13244 raft_consensus.cc:729] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.657234 13244 consensus_queue.cc:260] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.659507 13243 ts_tablet_manager.cc:1888] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.659662 13243 log.cc:1196] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/0254b176e3c24a818ffdd384a4bbcf00
I20240506 19:56:33.660115 13243 ts_tablet_manager.cc:1909] T 0254b176e3c24a818ffdd384a4bbcf00 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.661387 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 0254b176e3c24a818ffdd384a4bbcf00 (table default.Table_f9c09ec116c0499eace352aee64c8760 [id=d399bfc7bfab4572a933a4a2838b2ab4]) successfully deleted
W20240506 19:56:33.662281 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8596757ef75e433fb801f7679db5fefb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8596757ef75e433fb801f7679db5fefb already in progress: creating tablet
W20240506 19:56:33.668505 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81ea09efd4aa4019911dd23774f464a2 consensus state has no leader
I20240506 19:56:33.669801 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 0ca44a4cb8344307b94c8fafba143836 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.673496 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d5513f4926e41a6b78f02a5126cc13a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.673971 13244 ts_tablet_manager.cc:1398] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240506 19:56:33.674361 13244 tablet_bootstrap.cc:492] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:33.676174 13244 tablet_bootstrap.cc:654] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.687784 13246 ts_tablet_manager.cc:1888] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.688059 13246 log.cc:1196] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/3622008bbcf94ebcb2c2b78c7237b05a
I20240506 19:56:33.688562 13246 ts_tablet_manager.cc:1909] T 3622008bbcf94ebcb2c2b78c7237b05a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.689597 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet fbe1c6ed89d643ecb4e97402d78167b6
I20240506 19:56:33.689770 13246 tablet_replica.cc:330] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.689945 13246 raft_consensus.cc:2229] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:33.690148 12516 catalog_manager.cc:6554] ReplaceTablet: tablet fbe1c6ed89d643ecb4e97402d78167b6 deleted and replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:33.690318 13246 pending_rounds.cc:62] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:33.690424 13246 pending_rounds.cc:68] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744011850514432 op_type: NO_OP noop_request { }
I20240506 19:56:33.690584 13246 raft_consensus.cc:2866] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:33.690676 13246 raft_consensus.cc:2258] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.690831 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.693038 13245 tablet_bootstrap.cc:492] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.693204 13245 ts_tablet_manager.cc:1367] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.095s user 0.017s sys 0.016s
W20240506 19:56:33.693259 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet fbe1c6ed89d643ecb4e97402d78167b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbe1c6ed89d643ecb4e97402d78167b6 already in progress: creating tablet
I20240506 19:56:33.693997 13245 raft_consensus.cc:348] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.694216 13245 raft_consensus.cc:374] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.694289 13245 raft_consensus.cc:729] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.694530 13245 consensus_queue.cc:260] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.697562 13256 raft_consensus.cc:482] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.697760 13256 raft_consensus.cc:504] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.698347 13256 leader_election.cc:288] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:33.698448 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.699023 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:33.699350 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet fbe1c6ed89d643ecb4e97402d78167b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbe1c6ed89d643ecb4e97402d78167b6 already in progress: creating tablet
I20240506 19:56:33.699920 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:33.700708 13057 leader_election.cc:341] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:33.700990 13058 leader_election.cc:341] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.701107 13058 leader_election.cc:302] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.701597 13256 raft_consensus.cc:2726] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:33.701664 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f017e635010040008be72b3813b5a4ed
I20240506 19:56:33.702123 13245 ts_tablet_manager.cc:1398] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:56:33.702353 12521 catalog_manager.cc:6554] ReplaceTablet: tablet f017e635010040008be72b3813b5a4ed deleted and replaced by tablet 4a080ccac6394210b0b24b65843f9d75
W20240506 19:56:33.702920 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8842523a97954e3f81ffd87530201d93 consensus state has no leader
W20240506 19:56:33.703449 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acd3fc62e9614cf3bde0202ee6832bb3 has no consensus state
I20240506 19:56:33.703691 13245 tablet_bootstrap.cc:492] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.704420 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet f017e635010040008be72b3813b5a4ed with delete_type TABLET_DATA_DELETED (replaced by tablet 4a080ccac6394210b0b24b65843f9d75) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:33.705175 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f017e635010040008be72b3813b5a4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f017e635010040008be72b3813b5a4ed already in progress: creating tablet
I20240506 19:56:33.705677 13245 tablet_bootstrap.cc:654] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.706192 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet f017e635010040008be72b3813b5a4ed with delete_type TABLET_DATA_DELETED (replaced by tablet 4a080ccac6394210b0b24b65843f9d75) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.708253 13245 tablet_bootstrap.cc:492] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:33.708246 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f017e635010040008be72b3813b5a4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f017e635010040008be72b3813b5a4ed already in progress: creating tablet
I20240506 19:56:33.708417 13245 ts_tablet_manager.cc:1367] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240506 19:56:33.709405 13249 tablet_bootstrap.cc:492] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.709357 13245 raft_consensus.cc:348] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.709520 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 44de7153689b4dbd80dae87d3ee48a83 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.709591 13245 raft_consensus.cc:374] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.709681 13245 raft_consensus.cc:729] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.709699 13249 ts_tablet_manager.cc:1367] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.023s
I20240506 19:56:33.709959 13245 consensus_queue.cc:260] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.710475 13249 raft_consensus.cc:348] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.710697 13249 raft_consensus.cc:374] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.710795 13249 raft_consensus.cc:729] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.711048 13249 consensus_queue.cc:260] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.711176 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dd14f50dc4aa41a297c6484b3d750ab0" } new_table_name: "default.Table_d08a9f7decde4bb69b28102c4b21a5bd" modify_external_catalogs: true
I20240506 19:56:33.711771 13245 ts_tablet_manager.cc:1398] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:33.712894 13245 tablet_bootstrap.cc:492] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.714314 13249 ts_tablet_manager.cc:1398] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:33.714781 13249 tablet_bootstrap.cc:492] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.715588 13245 tablet_bootstrap.cc:654] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.716245 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b4a1ca7c489b432f93a40d7be8d66723" } modify_external_catalogs: true
I20240506 19:56:33.716511 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b4a1ca7c489b432f93a40d7be8d66723" } modify_external_catalogs: true
I20240506 19:56:33.716583 13249 tablet_bootstrap.cc:654] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.718542 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.719063 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet f017e635010040008be72b3813b5a4ed with delete_type TABLET_DATA_DELETED (replaced by tablet 4a080ccac6394210b0b24b65843f9d75) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.723657 13249 tablet_bootstrap.cc:492] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.723650 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 3622008bbcf94ebcb2c2b78c7237b05a (table default.Table_99963fe4c795413c98b34872a5ac9b5f [id=7a192bdbbc3b4571963f35dcd4b589d6]) successfully deleted
I20240506 19:56:33.723959 13249 ts_tablet_manager.cc:1367] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240506 19:56:33.724850 13249 raft_consensus.cc:348] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.725287 13249 raft_consensus.cc:374] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.725440 13249 raft_consensus.cc:729] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.725755 13249 consensus_queue.cc:260] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.726768 13249 ts_tablet_manager.cc:1398] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:33.727584 13249 tablet_bootstrap.cc:492] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.727824 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3d78abd7df2441f586f2f5aa221df837" } new_table_name: "default.Table_e16de993cdc24723b8a153e753457f59" modify_external_catalogs: true
I20240506 19:56:33.729440 13249 tablet_bootstrap.cc:654] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.732086 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1f7a4e714f4a4412a5843504c95b87d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.732735 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f7a4e714f4a4412a5843504c95b87d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.733829 13245 tablet_bootstrap.cc:492] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:33.734977 13249 tablet_bootstrap.cc:492] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.735147 13249 ts_tablet_manager.cc:1367] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240506 19:56:33.736075 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_62612e9478664e9d86a80129badd6c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.735999 13249 raft_consensus.cc:348] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.736349 13249 raft_consensus.cc:374] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.736443 13249 raft_consensus.cc:729] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:33.736682 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62612e9478664e9d86a80129badd6c12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.736707 13249 consensus_queue.cc:260] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.737944 13249 ts_tablet_manager.cc:1398] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:33.738562 13249 tablet_bootstrap.cc:492] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.742257 13249 tablet_bootstrap.cc:654] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.735594 13246 ts_tablet_manager.cc:1875] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.742700 13245 ts_tablet_manager.cc:1367] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20240506 19:56:33.743543 13245 raft_consensus.cc:348] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.746189 13245 raft_consensus.cc:374] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.746279 13245 raft_consensus.cc:729] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.746590 13245 consensus_queue.cc:260] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.747916 13256 raft_consensus.cc:482] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.748077 13256 raft_consensus.cc:504] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.748678 13256 leader_election.cc:288] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:33.749428 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e60cddf90fe4891be3c60651ebbb184" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:33.752568 13058 leader_election.cc:341] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.753609 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e60cddf90fe4891be3c60651ebbb184" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:33.753857 12988 raft_consensus.cc:2454] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:33.754405 13057 leader_election.cc:302] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.754979 13256 raft_consensus.cc:2781] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:33.755091 13256 raft_consensus.cc:482] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:33.755195 13256 raft_consensus.cc:3037] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.758348 13287 raft_consensus.cc:482] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.758554 13287 raft_consensus.cc:504] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.759249 13287 leader_election.cc:288] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:33.759913 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5513f4926e41a6b78f02a5126cc13a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:33.760486 12921 leader_election.cc:341] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.761328 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5513f4926e41a6b78f02a5126cc13a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:33.761574 13123 raft_consensus.cc:2454] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:33.762070 12922 leader_election.cc:302] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.762526 13287 raft_consensus.cc:2781] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:33.762609 13287 raft_consensus.cc:482] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:33.762708 13287 raft_consensus.cc:3037] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.765216 13249 tablet_bootstrap.cc:492] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.765429 13249 ts_tablet_manager.cc:1367] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.012s
W20240506 19:56:33.765583 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5e3f798d6484d4fabd55f9b4c98a911 consensus state has no leader
W20240506 19:56:33.766242 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b7f56fea34946e590dae0621b28f9c6 consensus state has no leader
I20240506 19:56:33.766183 13249 raft_consensus.cc:348] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.766417 13249 raft_consensus.cc:374] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.766491 13249 raft_consensus.cc:729] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.766783 13249 consensus_queue.cc:260] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.766830 13287 raft_consensus.cc:504] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.767187 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 1d5513f4926e41a6b78f02a5126cc13a (table default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 1 got 0
I20240506 19:56:33.767354 13249 ts_tablet_manager.cc:1398] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:33.768927 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5513f4926e41a6b78f02a5126cc13a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:33.769587 12921 leader_election.cc:341] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.770361 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5513f4926e41a6b78f02a5126cc13a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:33.770625 13123 raft_consensus.cc:3037] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.771343 13249 tablet_bootstrap.cc:492] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.775638 13249 tablet_bootstrap.cc:654] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.776868 12515 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 1d5513f4926e41a6b78f02a5126cc13a (table default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 1 got 0
I20240506 19:56:33.777223 12515 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 903ce1bc6ae7488d83bb0945424ed9f9 (table default.Table_675510c53e1848ed8176a861a1ace8c4 [id=2b7d2dfedf4d4169a2244e104dcd154d]). Expected version 3 got 0
I20240506 19:56:33.779436 13123 raft_consensus.cc:2454] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:33.779850 12826 tablet_service.cc:1467] Processing CreateTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 (DEFAULT_TABLE table=default.Table_dd14f50dc4aa41a297c6484b3d750ab0 [id=249eebb1852745929487a8fff7406ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.780035 12819 tablet_service.cc:1467] Processing CreateTablet for tablet acd3fc62e9614cf3bde0202ee6832bb3 (DEFAULT_TABLE table=default.Table_5bce626686754b56b9573ca9293b568f [id=c6e68683ddb542be99a54eea5e7a6ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.780331 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e3f798d6484d4fabd55f9b4c98a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.780591 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8a9ecf47d7474c82758417449d940c (DEFAULT_TABLE table=default.Table_8de44207890c4ceb8542f805a6f9b8da [id=70fc9e614eab4b11baeefbbf7127f540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.780632 12961 tablet_service.cc:1467] Processing CreateTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 (DEFAULT_TABLE table=default.Table_dd14f50dc4aa41a297c6484b3d750ab0 [id=249eebb1852745929487a8fff7406ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.780972 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8a9ecf47d7474c82758417449d940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.781015 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e3f798d6484d4fabd55f9b4c98a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.781597 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 393073bca2a04d15b3e583700262f3e0 (DEFAULT_TABLE table=default.Table_ddfa1d6244bd47dab51027f3945ed50f [id=fd3051399ebf4aa2b31bbd709b120d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.781927 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393073bca2a04d15b3e583700262f3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.782521 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.782866 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5516e8aad4e8474692b5b6c40bc483c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.785485 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.785637 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c (DEFAULT_TABLE table=default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.786057 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2502b423bd1040539fb1a3ebbece3c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.785996 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c (DEFAULT_TABLE table=default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.786136 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd3fc62e9614cf3bde0202ee6832bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.786341 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2502b423bd1040539fb1a3ebbece3c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.789696 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.790016 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8a9ecf47d7474c82758417449d940c (DEFAULT_TABLE table=default.Table_8de44207890c4ceb8542f805a6f9b8da [id=70fc9e614eab4b11baeefbbf7127f540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.790406 12955 tablet_service.cc:1467] Processing CreateTablet for tablet acd3fc62e9614cf3bde0202ee6832bb3 (DEFAULT_TABLE table=default.Table_5bce626686754b56b9573ca9293b568f [id=c6e68683ddb542be99a54eea5e7a6ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.790536 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 393073bca2a04d15b3e583700262f3e0 (DEFAULT_TABLE table=default.Table_ddfa1d6244bd47dab51027f3945ed50f [id=fd3051399ebf4aa2b31bbd709b120d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.790900 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca44a4cb8344307b94c8fafba143836 with delete_type TABLET_DATA_DELETED (replaced by tablet 92951843c8bd446aa86436b65d9eb793) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.791162 13243 tablet_replica.cc:330] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.791383 13243 raft_consensus.cc:2229] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.791555 13243 raft_consensus.cc:2258] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.791740 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5516e8aad4e8474692b5b6c40bc483c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.792286 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393073bca2a04d15b3e583700262f3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.792376 13243 ts_tablet_manager.cc:1875] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.791162 13249 tablet_bootstrap.cc:492] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.792760 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd3fc62e9614cf3bde0202ee6832bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.792856 13249 ts_tablet_manager.cc:1367] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240506 19:56:33.793213 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8a9ecf47d7474c82758417449d940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.794737 13249 raft_consensus.cc:348] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.794960 13249 raft_consensus.cc:374] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:33.795444 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0254b176e3c24a818ffdd384a4bbcf00 because the tablet was not found. No further retry: Not found: Tablet not found: 0254b176e3c24a818ffdd384a4bbcf00
I20240506 19:56:33.796110 12922 leader_election.cc:302] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.797840 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254b176e3c24a818ffdd384a4bbcf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.798153 13242 tablet_replica.cc:330] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:33.798364 13242 raft_consensus.cc:2229] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.798561 13242 raft_consensus.cc:2258] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.799397 13242 ts_tablet_manager.cc:1875] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.801609 13256 raft_consensus.cc:504] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.805815 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.806262 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5516e8aad4e8474692b5b6c40bc483c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.812004 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fd62bbf3494d06adb3d0493674558f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.811971 13093 tablet_service.cc:1467] Processing CreateTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 (DEFAULT_TABLE table=default.Table_dd14f50dc4aa41a297c6484b3d750ab0 [id=249eebb1852745929487a8fff7406ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.812517 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e3f798d6484d4fabd55f9b4c98a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.813109 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e60cddf90fe4891be3c60651ebbb184" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:33.814085 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c (DEFAULT_TABLE table=default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.814508 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2502b423bd1040539fb1a3ebbece3c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.817510 13329 raft_consensus.cc:482] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.817615 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e60cddf90fe4891be3c60651ebbb184" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:33.817672 13329 raft_consensus.cc:504] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.818491 13329 leader_election.cc:288] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:33.819629 13058 leader_election.cc:341] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.821599 12515 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_bedf0a0b6cd64f1bb0727feb78593f72 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 2 got 0
I20240506 19:56:33.822007 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2b51a8ac571347a896b8741066c330c4" } modify_external_catalogs: true
I20240506 19:56:33.822263 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2b51a8ac571347a896b8741066c330c4" } modify_external_catalogs: true
I20240506 19:56:33.825662 13246 ts_tablet_manager.cc:1888] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:33.825915 13246 log.cc:1196] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/75fd62bbf3494d06adb3d0493674558f
I20240506 19:56:33.826690 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162a5d3716c343139484a623105825a5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:33.827821 13058 leader_election.cc:341] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.831033 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d0e3260f46024386baac64cb8677883c
I20240506 19:56:33.831773 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1d5513f4926e41a6b78f02a5126cc13a
I20240506 19:56:33.832314 12522 catalog_manager.cc:6554] ReplaceTablet: tablet 1d5513f4926e41a6b78f02a5126cc13a deleted and replaced by tablet 2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:33.834596 13243 ts_tablet_manager.cc:1888] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.836407 13256 leader_election.cc:288] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:33.836551 13096 tablet_service.cc:1467] Processing CreateTablet for tablet acd3fc62e9614cf3bde0202ee6832bb3 (DEFAULT_TABLE table=default.Table_5bce626686754b56b9573ca9293b568f [id=c6e68683ddb542be99a54eea5e7a6ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.836983 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd3fc62e9614cf3bde0202ee6832bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.838678 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 162a5d3716c343139484a623105825a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.839015 13249 raft_consensus.cc:729] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.839041 13311 raft_consensus.cc:2781] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:33.839126 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e3260f46024386baac64cb8677883c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.839368 13249 consensus_queue.cc:260] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.840741 12988 raft_consensus.cc:3037] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.840934 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a80834187f874661b611b43065eee943
I20240506 19:56:33.841419 13243 log.cc:1196] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/0ca44a4cb8344307b94c8fafba143836
I20240506 19:56:33.841387 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162a5d3716c343139484a623105825a5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:33.842096 13057 leader_election.cc:341] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.842239 13057 leader_election.cc:302] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.842696 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet a80834187f874661b611b43065eee943 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.843087 13243 ts_tablet_manager.cc:1909] T 0ca44a4cb8344307b94c8fafba143836 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.844046 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a with delete_type TABLET_DATA_DELETED (replaced by tablet 2751074a54ed4bcea6979e6ad11c2efe) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.844110 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 0ca44a4cb8344307b94c8fafba143836 (table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]) successfully deleted
I20240506 19:56:33.826417 13246 ts_tablet_manager.cc:1909] T 75fd62bbf3494d06adb3d0493674558f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:33.844780 12988 raft_consensus.cc:2454] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
W20240506 19:56:33.844816 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d0e3260f46024386baac64cb8677883c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0e3260f46024386baac64cb8677883c already in progress: creating tablet
I20240506 19:56:33.845429 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8a9ecf47d7474c82758417449d940c (DEFAULT_TABLE table=default.Table_8de44207890c4ceb8542f805a6f9b8da [id=70fc9e614eab4b11baeefbbf7127f540]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:33.845525 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a80834187f874661b611b43065eee943 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a80834187f874661b611b43065eee943 already in progress: creating tablet
I20240506 19:56:33.845887 13243 tablet_replica.cc:330] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.846163 13246 tablet_replica.cc:330] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:33.846356 13246 raft_consensus.cc:2229] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.846541 13246 raft_consensus.cc:2258] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.849709 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e3260f46024386baac64cb8677883c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.853639 13287 leader_election.cc:288] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:33.853688 13057 leader_election.cc:302] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.853881 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a (DEFAULT_TABLE table=default.Table_1f7a4e714f4a4412a5843504c95b87d7 [id=9844f952118d4225a5a7489527a6a703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.854285 13329 raft_consensus.cc:2781] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:33.854369 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b24fa4663244a88dac9ea99fe38f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.854492 13329 raft_consensus.cc:686] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:33.854768 13329 consensus_queue.cc:237] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:33.856057 13242 ts_tablet_manager.cc:1888] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.856221 13244 tablet_bootstrap.cc:492] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.856566 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a with delete_type TABLET_DATA_DELETED (replaced by tablet 2751074a54ed4bcea6979e6ad11c2efe) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.857184 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a (DEFAULT_TABLE table=default.Table_1f7a4e714f4a4412a5843504c95b87d7 [id=9844f952118d4225a5a7489527a6a703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.859438 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 393073bca2a04d15b3e583700262f3e0 (DEFAULT_TABLE table=default.Table_ddfa1d6244bd47dab51027f3945ed50f [id=fd3051399ebf4aa2b31bbd709b120d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.861650 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 30d819d1237f4798b4847fdf378cbc62 (DEFAULT_TABLE table=default.Table_9d121b9e240b43cfbfea349bf45ef3a7 [id=332ee4f1670e4ddba0deb90d49e7b964]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.861819 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0594be752cc441b1af65142e6d925488" } modify_external_catalogs: true
I20240506 19:56:33.862074 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0594be752cc441b1af65142e6d925488" } modify_external_catalogs: true
I20240506 19:56:33.863073 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 30d819d1237f4798b4847fdf378cbc62 (DEFAULT_TABLE table=default.Table_9d121b9e240b43cfbfea349bf45ef3a7 [id=332ee4f1670e4ddba0deb90d49e7b964]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.863602 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d819d1237f4798b4847fdf378cbc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.864326 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5f69164a5a01441fa6dc72d34b878eb2" } modify_external_catalogs: true
I20240506 19:56:33.864570 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5f69164a5a01441fa6dc72d34b878eb2" } modify_external_catalogs: true
I20240506 19:56:33.864888 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_47a20691cc45428695761f4e7ef2e4fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.865686 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47a20691cc45428695761f4e7ef2e4fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.866106 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2b36704422c0483db4db7ab48455482d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.866677 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b36704422c0483db4db7ab48455482d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.867758 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_79cba7e264ba4a9e8b17244167924764"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.868224 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79cba7e264ba4a9e8b17244167924764 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.868027 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet a80834187f874661b611b43065eee943 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.869325 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b24fa4663244a88dac9ea99fe38f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.870103 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dfd40b6029db498e870d3e8a44ae6bdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.870589 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfd40b6029db498e870d3e8a44ae6bdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.872151 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fcd8a4763edf472692c23bb235cb7852" } modify_external_catalogs: true
I20240506 19:56:33.872115 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_32ef24b21b0a4018b9baabda7d2e552b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.872345 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fcd8a4763edf472692c23bb235cb7852" } modify_external_catalogs: true
I20240506 19:56:33.872416 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet c51b52a98cec48fc959b01a2a0a8795c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:33.872603 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32ef24b21b0a4018b9baabda7d2e552b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.873623 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 with delete_type TABLET_DATA_DELETED (replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.873836 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d819d1237f4798b4847fdf378cbc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.874512 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e with delete_type TABLET_DATA_DELETED (replaced by tablet a16704567ef6454eaef886fb9cd599ce) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.874130 13245 ts_tablet_manager.cc:1398] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.131s user 0.002s sys 0.000s
I20240506 19:56:33.876513 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e3260f46024386baac64cb8677883c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.877368 13242 log.cc:1196] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/0254b176e3c24a818ffdd384a4bbcf00
I20240506 19:56:33.877786 13242 ts_tablet_manager.cc:1909] T 0254b176e3c24a818ffdd384a4bbcf00 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:33.878399 13245 tablet_bootstrap.cc:492] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:33.878746 13242 tablet_replica.cc:330] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:33.878926 13242 raft_consensus.cc:2229] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.879107 13242 raft_consensus.cc:2258] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.879731 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 0254b176e3c24a818ffdd384a4bbcf00 (table default.Table_f9c09ec116c0499eace352aee64c8760 [id=d399bfc7bfab4572a933a4a2838b2ab4]) successfully deleted
I20240506 19:56:33.880322 13242 ts_tablet_manager.cc:1875] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.881793 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a with delete_type TABLET_DATA_DELETED (replaced by tablet 2751074a54ed4bcea6979e6ad11c2efe) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.882028 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet a80834187f874661b611b43065eee943 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.882591 13249 ts_tablet_manager.cc:1398] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.090s user 0.001s sys 0.005s
I20240506 19:56:33.883117 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a (DEFAULT_TABLE table=default.Table_1f7a4e714f4a4412a5843504c95b87d7 [id=9844f952118d4225a5a7489527a6a703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.889945 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 (DEFAULT_TABLE table=default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.890501 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a3bed0eb234e21b890242e9d23a5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.892715 13311 raft_consensus.cc:686] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:33.893075 13244 ts_tablet_manager.cc:1367] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.182s user 0.044s sys 0.026s
I20240506 19:56:33.893659 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet c51b52a98cec48fc959b01a2a0a8795c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.893934 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 with delete_type TABLET_DATA_DELETED (replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.894182 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e with delete_type TABLET_DATA_DELETED (replaced by tablet a16704567ef6454eaef886fb9cd599ce) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.893757 13311 consensus_queue.cc:237] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.894382 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet b63857fa6004468cad193ed4f2a9d366 with delete_type TABLET_DATA_DELETED (replaced by tablet 64b7e42061cd487b8517447eb59ea8a1) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.894585 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.894845 13249 tablet_bootstrap.cc:492] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.895092 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 (DEFAULT_TABLE table=default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.895534 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a3bed0eb234e21b890242e9d23a5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.899899 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 (DEFAULT_TABLE table=default.Table_f1f139bb215b41778590c78acb73a3d8 [id=bf75c039ad324e50a2ccb889439e2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.900103 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 (DEFAULT_TABLE table=default.Table_d91e484ef0cc4adb86af57d549a7e641 [id=9ad0be2302b94351876dba7a918a8d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.900449 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f66f1e46cce47639fe644fb4ed2e5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.900512 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ece0ad4df247eb9d380ed1c755ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.900514 13245 tablet_bootstrap.cc:654] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.902325 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 (DEFAULT_TABLE table=default.Table_f1f139bb215b41778590c78acb73a3d8 [id=bf75c039ad324e50a2ccb889439e2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.902626 13246 ts_tablet_manager.cc:1875] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.902709 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f66f1e46cce47639fe644fb4ed2e5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.902688 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba2365d00044c32a06f9017a207adf7 (DEFAULT_TABLE table=default.Table_8de45342e9fc4eaa8efcb1c4d73e818a [id=7a329c37e7684bac8bf008b1c5e53e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.903079 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba2365d00044c32a06f9017a207adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.903429 13249 tablet_bootstrap.cc:654] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.903276 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 75fd62bbf3494d06adb3d0493674558f (table default.Table_9439def23f6b4eaa8554197ebb5630a8 [id=a37fddd6fdbb47de9ac82b799503a40d]) successfully deleted
W20240506 19:56:33.903947 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8596757ef75e433fb801f7679db5fefb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8596757ef75e433fb801f7679db5fefb already in progress: creating tablet
W20240506 19:56:33.904458 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet fbe1c6ed89d643ecb4e97402d78167b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbe1c6ed89d643ecb4e97402d78167b6 already in progress: creating tablet
I20240506 19:56:33.904826 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 4a080ccac6394210b0b24b65843f9d75 (DEFAULT_TABLE table=default.Table_c715baa1d25447d991935176e0f72c6b [id=4b9a2f4ea4b1460f89e8981e952e0d2d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:33.905043 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f017e635010040008be72b3813b5a4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f017e635010040008be72b3813b5a4ed already in progress: creating tablet
I20240506 19:56:33.905182 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a080ccac6394210b0b24b65843f9d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.905337 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 4a080ccac6394210b0b24b65843f9d75 (DEFAULT_TABLE table=default.Table_c715baa1d25447d991935176e0f72c6b [id=4b9a2f4ea4b1460f89e8981e952e0d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.905716 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a080ccac6394210b0b24b65843f9d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.906378 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 30d819d1237f4798b4847fdf378cbc62 (DEFAULT_TABLE table=default.Table_9d121b9e240b43cfbfea349bf45ef3a7 [id=332ee4f1670e4ddba0deb90d49e7b964]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.911309 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8842523a97954e3f81ffd87530201d93
I20240506 19:56:33.893975 13244 raft_consensus.cc:348] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.912240 13244 raft_consensus.cc:374] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.912341 13244 raft_consensus.cc:729] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.912652 13244 consensus_queue.cc:260] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.913225 13244 ts_tablet_manager.cc:1398] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240506 19:56:33.913642 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 (DEFAULT_TABLE table=default.Table_d91e484ef0cc4adb86af57d549a7e641 [id=9ad0be2302b94351876dba7a918a8d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.913702 13329 raft_consensus.cc:482] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.913872 13329 raft_consensus.cc:504] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.914068 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8a9ecf47d7474c82758417449d940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.914616 13329 leader_election.cc:288] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:33.915865 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51b52a98cec48fc959b01a2a0a8795c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:33.916118 12988 raft_consensus.cc:2454] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:33.916700 13057 leader_election.cc:302] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:33.917021 13329 raft_consensus.cc:2781] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:33.917130 13329 raft_consensus.cc:482] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:33.917210 13329 raft_consensus.cc:3037] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:33.917402 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d819d1237f4798b4847fdf378cbc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.919344 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 8842523a97954e3f81ffd87530201d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.924665 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 8842523a97954e3f81ffd87530201d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.926126 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 79486583025c42bfb1321f9cb6dd96f1
I20240506 19:56:33.927889 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:33.928381 13243 raft_consensus.cc:2229] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:33.928941 13243 pending_rounds.cc:62] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:33.929047 13243 pending_rounds.cc:68] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744013396295680 op_type: NO_OP noop_request { }
I20240506 19:56:33.929214 13243 raft_consensus.cc:2866] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:33.929395 13243 raft_consensus.cc:2258] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:33.930752 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba2365d00044c32a06f9017a207adf7 (DEFAULT_TABLE table=default.Table_8de45342e9fc4eaa8efcb1c4d73e818a [id=7a329c37e7684bac8bf008b1c5e53e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.933193 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.933633 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51b52a98cec48fc959b01a2a0a8795c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:33.937973 13244 tablet_bootstrap.cc:492] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:33.939792 13244 tablet_bootstrap.cc:654] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:33.941653 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ece0ad4df247eb9d380ed1c755ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.947254 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c905679471c7485fa59180f110b5ec1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.947903 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c905679471c7485fa59180f110b5ec1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.949368 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f1f139bb215b41778590c78acb73a3d8" } new_table_name: "default.Table_92531c4a95d14b50b186416c9cfd94b8" modify_external_catalogs: true
I20240506 19:56:33.950137 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 903ce1bc6ae7488d83bb0945424ed9f9 (table default.Table_675510c53e1848ed8176a861a1ace8c4 [id=2b7d2dfedf4d4169a2244e104dcd154d]). Expected version 3 got 0
I20240506 19:56:33.952189 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 393073bca2a04d15b3e583700262f3e0 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:33.954020 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3322d5e06d7f437faf751479f7bd019f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:33.954552 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4b6f407e4a464024aed5a1fb84c49849"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:33.955394 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b6f407e4a464024aed5a1fb84c49849 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:33.956199 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3322d5e06d7f437faf751479f7bd019f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:33.956861 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1af9d86b2ea64c83914a6ae32457b2e0" } new_table_name: "default.Table_3a86534bf1014d9890190f09491e899d" modify_external_catalogs: true
I20240506 19:56:33.957803 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 598998d27cc44e6a9c8cf4ccd5a2b00a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:33.958109 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e16de993cdc24723b8a153e753457f59" } new_table_name: "default.Table_65407acf1df5470f8ec9e008fdbcbd93" modify_external_catalogs: true
I20240506 19:56:33.931902 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba2365d00044c32a06f9017a207adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.940145 13311 raft_consensus.cc:482] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.960268 13311 raft_consensus.cc:504] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.961042 13311 leader_election.cc:288] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:33.961761 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63857fa6004468cad193ed4f2a9d366" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:33.962512 12921 leader_election.cc:341] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.960019 13249 tablet_bootstrap.cc:492] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:33.963152 13249 ts_tablet_manager.cc:1367] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.002s
W20240506 19:56:33.940488 13356 consensus_queue.cc:1159] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "059a5c4a3f7e49ef95af4b8341702745" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240506 19:56:33.964027 13249 raft_consensus.cc:348] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.964171 13243 ts_tablet_manager.cc:1875] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:33.964437 13249 raft_consensus.cc:374] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.964557 13249 raft_consensus.cc:729] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.964871 13249 consensus_queue.cc:260] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:33.965602 13249 ts_tablet_manager.cc:1398] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:33.965938 13249 tablet_bootstrap.cc:492] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:33.967592 13249 tablet_bootstrap.cc:654] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:33.977135 13058 leader_election.cc:341] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.978562 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63857fa6004468cad193ed4f2a9d366" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:33.978680 13244 tablet_bootstrap.cc:492] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:33.978832 13244 ts_tablet_manager.cc:1367] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.001s
I20240506 19:56:33.978945 13242 ts_tablet_manager.cc:1888] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:33.979075 13242 log.cc:1196] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/75fd62bbf3494d06adb3d0493674558f
I20240506 19:56:33.979475 13242 ts_tablet_manager.cc:1909] T 75fd62bbf3494d06adb3d0493674558f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:33.979681 12922 leader_election.cc:341] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:33.979869 12922 leader_election.cc:302] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:33.980230 13311 raft_consensus.cc:2726] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:33.980978 13243 ts_tablet_manager.cc:1888] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:33.979740 13244 raft_consensus.cc:348] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.981344 13244 raft_consensus.cc:374] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:33.981372 13243 log.cc:1196] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1d5513f4926e41a6b78f02a5126cc13a
I20240506 19:56:33.981602 13244 raft_consensus.cc:729] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:33.982594 13243 ts_tablet_manager.cc:1909] T 1d5513f4926e41a6b78f02a5126cc13a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:33.982985 13244 consensus_queue.cc:260] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.983605 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 (DEFAULT_TABLE table=default.Table_d91e484ef0cc4adb86af57d549a7e641 [id=9ad0be2302b94351876dba7a918a8d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.984130 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b24fa4663244a88dac9ea99fe38f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.985936 13256 raft_consensus.cc:482] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:33.986344 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 (DEFAULT_TABLE table=default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.987075 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 (DEFAULT_TABLE table=default.Table_f1f139bb215b41778590c78acb73a3d8 [id=bf75c039ad324e50a2ccb889439e2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.987263 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 75fd62bbf3494d06adb3d0493674558f (table default.Table_9439def23f6b4eaa8554197ebb5630a8 [id=a37fddd6fdbb47de9ac82b799503a40d]) successfully deleted
I20240506 19:56:33.987850 13244 ts_tablet_manager.cc:1398] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:56:33.988013 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ece0ad4df247eb9d380ed1c755ac99. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:33.988464 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d0e3260f46024386baac64cb8677883c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0e3260f46024386baac64cb8677883c already in progress: creating tablet
W20240506 19:56:33.989118 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1d5513f4926e41a6b78f02a5126cc13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d5513f4926e41a6b78f02a5126cc13a already in progress: creating tablet
I20240506 19:56:33.989406 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 4a080ccac6394210b0b24b65843f9d75 (DEFAULT_TABLE table=default.Table_c715baa1d25447d991935176e0f72c6b [id=4b9a2f4ea4b1460f89e8981e952e0d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.989486 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1d5513f4926e41a6b78f02a5126cc13a (table default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]) successfully deleted
I20240506 19:56:33.989732 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 8842523a97954e3f81ffd87530201d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:33.990195 13243 tablet_replica.cc:330] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:33.990658 13243 raft_consensus.cc:2229] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:33.991683 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f66f1e46cce47639fe644fb4ed2e5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.991160 13256 raft_consensus.cc:504] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:33.991037 13243 raft_consensus.cc:2258] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:33.993857 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a80834187f874661b611b43065eee943 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a80834187f874661b611b43065eee943 already in progress: creating tablet
I20240506 19:56:33.993921 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdcdbb9cf3e4e9db6174bce7785565e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:33.994943 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdcdbb9cf3e4e9db6174bce7785565e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:33.995164 12988 raft_consensus.cc:2454] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:33.995555 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba2365d00044c32a06f9017a207adf7 (DEFAULT_TABLE table=default.Table_8de45342e9fc4eaa8efcb1c4d73e818a [id=7a329c37e7684bac8bf008b1c5e53e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:33.995875 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a3bed0eb234e21b890242e9d23a5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.996119 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a080ccac6394210b0b24b65843f9d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.997594 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393073bca2a04d15b3e583700262f3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:33.997685 13244 tablet_bootstrap.cc:492] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:33.998662 13256 leader_election.cc:288] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:33.999166 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:33.999289 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8842523a97954e3f81ffd87530201d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8842523a97954e3f81ffd87530201d93 already in progress: creating tablet
W20240506 19:56:33.999922 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 79486583025c42bfb1321f9cb6dd96f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79486583025c42bfb1321f9cb6dd96f1 already in progress: creating tablet
W20240506 19:56:34.000193 13058 leader_election.cc:341] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.000241 13242 tablet_replica.cc:330] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:34.000437 13242 raft_consensus.cc:2229] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.000675 13242 raft_consensus.cc:2258] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.001744 13244 tablet_bootstrap.cc:654] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.005668 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba2365d00044c32a06f9017a207adf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.005967 13243 ts_tablet_manager.cc:1875] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.007140 13242 ts_tablet_manager.cc:1875] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.012069 13244 tablet_bootstrap.cc:492] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:34.012279 13244 ts_tablet_manager.cc:1367] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240506 19:56:34.013056 13244 raft_consensus.cc:348] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.013362 13244 raft_consensus.cc:374] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.013449 13244 raft_consensus.cc:729] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.014611 13244 consensus_queue.cc:260] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.015074 13329 raft_consensus.cc:504] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.015247 13244 ts_tablet_manager.cc:1398] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:34.015568 13244 tablet_bootstrap.cc:492] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:34.015843 13329 leader_election.cc:288] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:34.016165 12506 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 64b7e42061cd487b8517447eb59ea8a1 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 64b7e42061cd487b8517447eb59ea8a1 consensus state has no leader
I20240506 19:56:34.017426 13244 tablet_bootstrap.cc:654] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.017669 13243 ts_tablet_manager.cc:1888] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.017655 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51b52a98cec48fc959b01a2a0a8795c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:34.017791 13243 log.cc:1196] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/c51b52a98cec48fc959b01a2a0a8795c
I20240506 19:56:34.018179 13243 ts_tablet_manager.cc:1909] T c51b52a98cec48fc959b01a2a0a8795c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.019276 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet c51b52a98cec48fc959b01a2a0a8795c (table default.Table_2a4c70f40666495ba552107168859093 [id=513f5de1c20e4aa48bd9f216c869b5fe]) successfully deleted
W20240506 19:56:34.019316 13058 leader_election.cc:341] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.020473 13243 tablet_replica.cc:330] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:34.020743 13243 raft_consensus.cc:2229] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.021174 13243 raft_consensus.cc:2258] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.021382 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 18920e08b0de4f5292abb54e3432ee39
I20240506 19:56:34.021664 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e60cddf90fe4891be3c60651ebbb184 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.022068 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.022136 13243 ts_tablet_manager.cc:1875] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.023029 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 18920e08b0de4f5292abb54e3432ee39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.025141 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_bedf0a0b6cd64f1bb0727feb78593f72 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 2 got 0
I20240506 19:56:34.027004 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 18920e08b0de4f5292abb54e3432ee39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:34.028113 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f66f1e46cce47639fe644fb4ed2e5c0 consensus state has no leader
I20240506 19:56:34.029635 13243 ts_tablet_manager.cc:1888] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.029809 13243 log.cc:1196] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/6e60cddf90fe4891be3c60651ebbb184
I20240506 19:56:34.030189 13243 ts_tablet_manager.cc:1909] T 6e60cddf90fe4891be3c60651ebbb184 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.031337 13057 leader_election.cc:302] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.031486 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 6e60cddf90fe4891be3c60651ebbb184 (table default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:34.031795 13329 raft_consensus.cc:2781] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:34.031911 13329 raft_consensus.cc:482] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:34.032037 13329 raft_consensus.cc:3037] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:34.032845 13057 leader_election.cc:341] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: c51b52a98cec48fc959b01a2a0a8795c
I20240506 19:56:34.033010 13057 leader_election.cc:302] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.033483 13256 raft_consensus.cc:2726] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:34.033658 13245 tablet_bootstrap.cc:492] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.033685 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622008bbcf94ebcb2c2b78c7237b05a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.033813 13245 ts_tablet_manager.cc:1367] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.010s
I20240506 19:56:34.033685 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 162a5d3716c343139484a623105825a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.034706 13245 raft_consensus.cc:348] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.034929 13245 raft_consensus.cc:374] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.035035 13245 raft_consensus.cc:729] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.034958 13329 raft_consensus.cc:504] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.035279 13245 consensus_queue.cc:260] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.036176 13245 ts_tablet_manager.cc:1398] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:34.039275 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bedf0a0b6cd64f1bb0727feb78593f72" } new_table_name: "default.Table_7af138f7bd89452c9f067f9c2fc6f955" modify_external_catalogs: true
I20240506 19:56:34.040738 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d90b50896a6b4231a2564273cefacca8" } new_table_name: "default.Table_bd219422c6e141dea81c43e0227f351c" modify_external_catalogs: true
I20240506 19:56:34.045259 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdcdbb9cf3e4e9db6174bce7785565e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:34.045542 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 18920e08b0de4f5292abb54e3432ee39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:34.051000 13058 leader_election.cc:341] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.051998 13243 tablet_replica.cc:330] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:34.052017 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdcdbb9cf3e4e9db6174bce7785565e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:34.052275 12988 raft_consensus.cc:3037] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:34.054010 13329 leader_election.cc:288] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:34.054658 12988 raft_consensus.cc:2454] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:34.055073 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_60bc98ea5883422dac4600f737dc29f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.055706 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60bc98ea5883422dac4600f737dc29f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.056862 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 162a5d3716c343139484a623105825a5 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.062510 13245 tablet_bootstrap.cc:492] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.065733 13249 tablet_bootstrap.cc:492] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:34.065917 13249 ts_tablet_manager.cc:1367] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.004s
I20240506 19:56:34.066468 13243 raft_consensus.cc:2229] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.066823 13249 raft_consensus.cc:348] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.067026 13249 raft_consensus.cc:374] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.067102 13249 raft_consensus.cc:729] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.067320 13249 consensus_queue.cc:260] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.067854 13249 ts_tablet_manager.cc:1398] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:34.068171 13249 tablet_bootstrap.cc:492] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:34.070065 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 393073bca2a04d15b3e583700262f3e0
I20240506 19:56:34.070561 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 393073bca2a04d15b3e583700262f3e0 deleted and replaced by tablet b529bb9362924caca399491e7d9e03cd
I20240506 19:56:34.070668 13249 tablet_bootstrap.cc:654] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.066169 13057 leader_election.cc:302] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.071552 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e6c606a634df58f2180361078d3d5 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e3260f46024386baac64cb8677883c) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.072463 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 393073bca2a04d15b3e583700262f3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b529bb9362924caca399491e7d9e03cd) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.073422 13329 raft_consensus.cc:2781] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:34.074010 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 393073bca2a04d15b3e583700262f3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b529bb9362924caca399491e7d9e03cd) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.074952 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 393073bca2a04d15b3e583700262f3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b529bb9362924caca399491e7d9e03cd) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.075539 13329 raft_consensus.cc:686] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:34.075870 13329 consensus_queue.cc:237] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.079980 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3501606fb8de42f68fd7894fa92e1a95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.082044 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3501606fb8de42f68fd7894fa92e1a95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.082263 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e6c606a634df58f2180361078d3d5 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e3260f46024386baac64cb8677883c) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.089915 13311 raft_consensus.cc:482] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.090126 13311 raft_consensus.cc:504] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.090806 13311 leader_election.cc:288] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:34.091392 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2abecd9ec7e4606b38c7d22477ddf65" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:34.092219 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8de44207890c4ceb8542f805a6f9b8da" } modify_external_catalogs: true
I20240506 19:56:34.092177 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9d121b9e240b43cfbfea349bf45ef3a7" } modify_external_catalogs: true
I20240506 19:56:34.092428 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8de44207890c4ceb8542f805a6f9b8da" } modify_external_catalogs: true
W20240506 19:56:34.092551 12921 leader_election.cc:341] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.095487 13242 ts_tablet_manager.cc:1888] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.095634 13242 log.cc:1196] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/598998d27cc44e6a9c8cf4ccd5a2b00a
I20240506 19:56:34.095963 13242 ts_tablet_manager.cc:1909] T 598998d27cc44e6a9c8cf4ccd5a2b00a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:34.096541 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 598998d27cc44e6a9c8cf4ccd5a2b00a (table default.Table_2beea19b5bb94396b6941f206811de8c [id=8c7219b61355477582e0d6d88b0ce18f]) successfully deleted
I20240506 19:56:34.097909 13252 raft_consensus.cc:482] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.098089 13252 raft_consensus.cc:504] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.098798 13252 leader_election.cc:288] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:34.099396 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9d121b9e240b43cfbfea349bf45ef3a7" } modify_external_catalogs: true
I20240506 19:56:34.100963 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162a5d3716c343139484a623105825a5" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:34.101325 13246 ts_tablet_manager.cc:1888] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.101508 13246 log.cc:1196] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/162a5d3716c343139484a623105825a5
I20240506 19:56:34.101958 13246 ts_tablet_manager.cc:1909] T 162a5d3716c343139484a623105825a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.102309 13243 raft_consensus.cc:2258] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:34.102353 12781 leader_election.cc:341] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.103060 13246 tablet_replica.cc:330] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.103276 13246 raft_consensus.cc:2229] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.103425 13243 ts_tablet_manager.cc:1875] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.103446 13246 raft_consensus.cc:2258] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.104615 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 162a5d3716c343139484a623105825a5 (table default.Table_d36494da319a48e18299fbb1708e484b [id=d1e6677412624f19917aae666e3e5184]) successfully deleted
I20240506 19:56:34.105149 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2abecd9ec7e4606b38c7d22477ddf65" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:34.105693 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 18920e08b0de4f5292abb54e3432ee39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18920e08b0de4f5292abb54e3432ee39 already in progress: creating tablet
W20240506 19:56:34.109019 12922 leader_election.cc:341] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.109174 12922 leader_election.cc:302] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.109901 13311 raft_consensus.cc:2726] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:34.109964 12783 leader_election.cc:341] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 162a5d3716c343139484a623105825a5
I20240506 19:56:34.110100 12783 leader_election.cc:302] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:34.110541 13252 raft_consensus.cc:2726] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:34.117928 12508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240506 19:56:34.118492 13242 tablet_replica.cc:330] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:34.119362 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a16704567ef6454eaef886fb9cd599ce consensus state has no leader
I20240506 19:56:34.119680 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f66f1e46cce47639fe644fb4ed2e5c0 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.119750 13242 raft_consensus.cc:2229] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.119956 13242 raft_consensus.cc:2258] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:34.120097 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b7f56fea34946e590dae0621b28f9c6 consensus state has no leader
W20240506 19:56:34.122061 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81ea09efd4aa4019911dd23774f464a2 consensus state has no leader
W20240506 19:56:34.122737 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6156c6f582b747de973b357b6d071a46 consensus state has no leader
I20240506 19:56:34.128259 12960 tablet_service.cc:1467] Processing CreateTablet for tablet b38453bf678748399339806d12993c88 (DEFAULT_TABLE table=default.Table_62612e9478664e9d86a80129badd6c12 [id=8f671c29345e43b6ba8379f8182cba65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.128762 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38453bf678748399339806d12993c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.131080 13249 tablet_bootstrap.cc:492] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:34.131428 13249 ts_tablet_manager.cc:1367] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.004s
I20240506 19:56:34.132802 13249 raft_consensus.cc:348] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.133231 13249 raft_consensus.cc:374] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.133410 13249 raft_consensus.cc:729] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.133659 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe (DEFAULT_TABLE table=default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.134119 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2751074a54ed4bcea6979e6ad11c2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.137516 12826 tablet_service.cc:1467] Processing CreateTablet for tablet b38453bf678748399339806d12993c88 (DEFAULT_TABLE table=default.Table_62612e9478664e9d86a80129badd6c12 [id=8f671c29345e43b6ba8379f8182cba65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.138006 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38453bf678748399339806d12993c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.140723 13242 ts_tablet_manager.cc:1875] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.140776 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe (DEFAULT_TABLE table=default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.141170 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2751074a54ed4bcea6979e6ad11c2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.144572 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 6d8a9ecf47d7474c82758417449d940c
I20240506 19:56:34.145562 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:34.146045 12520 catalog_manager.cc:6554] ReplaceTablet: tablet 8f66f1e46cce47639fe644fb4ed2e5c0 deleted and replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:34.145550 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8a9ecf47d7474c82758417449d940c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.147080 13244 tablet_bootstrap.cc:492] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:34.147256 13246 ts_tablet_manager.cc:1875] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.147375 13244 ts_tablet_manager.cc:1367] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.132s user 0.013s sys 0.022s
I20240506 19:56:34.150108 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8a9ecf47d7474c82758417449d940c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.152835 13244 raft_consensus.cc:348] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.153050 13244 raft_consensus.cc:374] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.153146 13244 raft_consensus.cc:729] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.153270 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe (DEFAULT_TABLE table=default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.153764 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2751074a54ed4bcea6979e6ad11c2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.154760 13244 consensus_queue.cc:260] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.155267 13244 ts_tablet_manager.cc:1398] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:56:34.156315 13244 tablet_bootstrap.cc:492] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:34.158023 13244 tablet_bootstrap.cc:654] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.160259 13093 tablet_service.cc:1467] Processing CreateTablet for tablet b38453bf678748399339806d12993c88 (DEFAULT_TABLE table=default.Table_62612e9478664e9d86a80129badd6c12 [id=8f671c29345e43b6ba8379f8182cba65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.166332 12962 tablet_service.cc:1467] Processing CreateTablet for tablet d380bf30599d4130bbb80b4f26837779 (DEFAULT_TABLE table=default.Table_47a20691cc45428695761f4e7ef2e4fc [id=873dfe33ce7141f2bc25d7575a63fc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.166803 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d380bf30599d4130bbb80b4f26837779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.167440 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1f7a4e714f4a4412a5843504c95b87d7" } modify_external_catalogs: true
I20240506 19:56:34.167677 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1f7a4e714f4a4412a5843504c95b87d7" } modify_external_catalogs: true
I20240506 19:56:34.167905 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.168280 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.134097 13249 consensus_queue.cc:260] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.169157 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1ba0602f38e64789a126e75fc115f5c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:34.169672 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 5988b89cb52e4f858b52862138bed025 (DEFAULT_TABLE table=default.Table_2b36704422c0483db4db7ab48455482d [id=cd2382012b444dd09cd9ad1a50a20b4c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:34.169786 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ba0602f38e64789a126e75fc115f5c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.170091 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5988b89cb52e4f858b52862138bed025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.171273 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e5d18039b7f04edaa0c985be5260be35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:34.172490 13249 ts_tablet_manager.cc:1398] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240506 19:56:34.172819 13249 tablet_bootstrap.cc:492] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:34.174827 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5d18039b7f04edaa0c985be5260be35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.175073 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b50238125ad14920bf820245406918c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:34.167435 12515 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:34.175721 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfd40b6029db498e870d3e8a44ae6bdd" } modify_external_catalogs: true
I20240506 19:56:34.175920 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfd40b6029db498e870d3e8a44ae6bdd" } modify_external_catalogs: true
I20240506 19:56:34.176604 13249 tablet_bootstrap.cc:654] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.178169 13243 ts_tablet_manager.cc:1888] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.178298 13243 log.cc:1196] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/7bdcdbb9cf3e4e9db6174bce7785565e
I20240506 19:56:34.178709 13243 ts_tablet_manager.cc:1909] T 7bdcdbb9cf3e4e9db6174bce7785565e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.178632 13101 tablet_service.cc:1467] Processing CreateTablet for tablet f3d57249f30a44dab3384092ec20b515 (DEFAULT_TABLE table=default.Table_32ef24b21b0a4018b9baabda7d2e552b [id=7197d7d0733346f4bc417c7dca54b79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.179284 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 4f91888e88234cbdbd00f48dc1574653 (DEFAULT_TABLE table=default.Table_4b6f407e4a464024aed5a1fb84c49849 [id=c7b8c2f7b9f74051b8207e0e127d930d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.179677 13243 tablet_replica.cc:330] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:34.179870 13243 raft_consensus.cc:2229] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.180033 13243 raft_consensus.cc:2258] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.180836 13243 ts_tablet_manager.cc:1875] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.181330 13311 raft_consensus.cc:482] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.181515 13311 raft_consensus.cc:504] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.182209 13311 leader_election.cc:288] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:34.182647 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:34.183338 12921 leader_election.cc:341] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.185060 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (DEFAULT_TABLE table=default.Table_3322d5e06d7f437faf751479f7bd019f [id=f23583e607a4419986d06a38c45bbdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.185109 12959 tablet_service.cc:1467] Processing CreateTablet for tablet ee59a76d52fc48d7bcf3256c359d3c02 (DEFAULT_TABLE table=default.Table_79cba7e264ba4a9e8b17244167924764 [id=d9b546ebb0c146a7920cc05c08a29e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.185365 12955 tablet_service.cc:1467] Processing CreateTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 (DEFAULT_TABLE table=default.Table_c905679471c7485fa59180f110b5ec1f [id=c4769761819944109eebf88cb916cb27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.185534 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.185720 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c656e16c1045dea79bc23b64fdd5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.185942 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee59a76d52fc48d7bcf3256c359d3c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.186224 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 4f91888e88234cbdbd00f48dc1574653 (DEFAULT_TABLE table=default.Table_4b6f407e4a464024aed5a1fb84c49849 [id=c7b8c2f7b9f74051b8207e0e127d930d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.186571 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f91888e88234cbdbd00f48dc1574653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.197229 13242 ts_tablet_manager.cc:1888] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.197724 13242 log.cc:1196] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/3622008bbcf94ebcb2c2b78c7237b05a
I20240506 19:56:34.197770 13243 ts_tablet_manager.cc:1888] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.197929 13243 log.cc:1196] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b63857fa6004468cad193ed4f2a9d366
I20240506 19:56:34.198299 13243 ts_tablet_manager.cc:1909] T b63857fa6004468cad193ed4f2a9d366 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.198781 13242 ts_tablet_manager.cc:1909] T 3622008bbcf94ebcb2c2b78c7237b05a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:34.199319 13243 tablet_replica.cc:330] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:34.199538 13243 raft_consensus.cc:2229] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.199724 13243 raft_consensus.cc:2258] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.200975 12821 tablet_service.cc:1467] Processing CreateTablet for tablet f3d57249f30a44dab3384092ec20b515 (DEFAULT_TABLE table=default.Table_32ef24b21b0a4018b9baabda7d2e552b [id=7197d7d0733346f4bc417c7dca54b79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.201481 12815 tablet_service.cc:1467] Processing CreateTablet for tablet ee59a76d52fc48d7bcf3256c359d3c02 (DEFAULT_TABLE table=default.Table_79cba7e264ba4a9e8b17244167924764 [id=d9b546ebb0c146a7920cc05c08a29e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.201613 12822 tablet_service.cc:1467] Processing CreateTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 (DEFAULT_TABLE table=default.Table_c905679471c7485fa59180f110b5ec1f [id=c4769761819944109eebf88cb916cb27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.201880 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 5988b89cb52e4f858b52862138bed025 (DEFAULT_TABLE table=default.Table_2b36704422c0483db4db7ab48455482d [id=cd2382012b444dd09cd9ad1a50a20b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.202181 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 4f91888e88234cbdbd00f48dc1574653 (DEFAULT_TABLE table=default.Table_4b6f407e4a464024aed5a1fb84c49849 [id=c7b8c2f7b9f74051b8207e0e127d930d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.202225 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 1667a0bbd3af4d678434b068c47cc518 (DEFAULT_TABLE table=default.Table_dfd40b6029db498e870d3e8a44ae6bdd [id=c4a20cff71d7496e8b07f477a0c1ea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.202400 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (DEFAULT_TABLE table=default.Table_3322d5e06d7f437faf751479f7bd019f [id=f23583e607a4419986d06a38c45bbdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.201965 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee59a76d52fc48d7bcf3256c359d3c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.202618 12817 tablet_service.cc:1467] Processing CreateTablet for tablet b529bb9362924caca399491e7d9e03cd (DEFAULT_TABLE table=default.Table_ddfa1d6244bd47dab51027f3945ed50f [id=fd3051399ebf4aa2b31bbd709b120d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.201488 12827 tablet_service.cc:1467] Processing CreateTablet for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (DEFAULT_TABLE table=default.Table_60bc98ea5883422dac4600f737dc29f1 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.203037 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.205693 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f1e4efbe8b40b199fcb3020569e1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.206163 13244 tablet_bootstrap.cc:492] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:34.206341 13244 ts_tablet_manager.cc:1367] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.000s
I20240506 19:56:34.206512 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5988b89cb52e4f858b52862138bed025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.207247 13244 raft_consensus.cc:348] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.207499 13244 raft_consensus.cc:374] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.207578 13244 raft_consensus.cc:729] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.208019 13245 tablet_bootstrap.cc:654] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.209175 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38453bf678748399339806d12993c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.209688 12960 tablet_service.cc:1467] Processing CreateTablet for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (DEFAULT_TABLE table=default.Table_60bc98ea5883422dac4600f737dc29f1 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.210117 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f1e4efbe8b40b199fcb3020569e1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.210479 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 5988b89cb52e4f858b52862138bed025 (DEFAULT_TABLE table=default.Table_2b36704422c0483db4db7ab48455482d [id=cd2382012b444dd09cd9ad1a50a20b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.212425 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:34.213304 13103 tablet_service.cc:1467] Processing CreateTablet for tablet ee59a76d52fc48d7bcf3256c359d3c02 (DEFAULT_TABLE table=default.Table_79cba7e264ba4a9e8b17244167924764 [id=d9b546ebb0c146a7920cc05c08a29e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.213965 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (DEFAULT_TABLE table=default.Table_3322d5e06d7f437faf751479f7bd019f [id=f23583e607a4419986d06a38c45bbdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.217459 12961 tablet_service.cc:1467] Processing CreateTablet for tablet f3d57249f30a44dab3384092ec20b515 (DEFAULT_TABLE table=default.Table_32ef24b21b0a4018b9baabda7d2e552b [id=7197d7d0733346f4bc417c7dca54b79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.217965 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d57249f30a44dab3384092ec20b515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.219242 13243 ts_tablet_manager.cc:1875] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.221159 13100 tablet_service.cc:1467] Processing CreateTablet for tablet d380bf30599d4130bbb80b4f26837779 (DEFAULT_TABLE table=default.Table_47a20691cc45428695761f4e7ef2e4fc [id=873dfe33ce7141f2bc25d7575a63fc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.221588 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.222092 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 1667a0bbd3af4d678434b068c47cc518 (DEFAULT_TABLE table=default.Table_dfd40b6029db498e870d3e8a44ae6bdd [id=c4a20cff71d7496e8b07f477a0c1ea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.222549 13091 tablet_service.cc:1467] Processing CreateTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 (DEFAULT_TABLE table=default.Table_c905679471c7485fa59180f110b5ec1f [id=c4769761819944109eebf88cb916cb27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.232632 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d57249f30a44dab3384092ec20b515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.237756 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8a9ecf47d7474c82758417449d940c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.238318 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c656e16c1045dea79bc23b64fdd5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.241708 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee59a76d52fc48d7bcf3256c359d3c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.243806 13311 raft_consensus.cc:482] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:34.245064 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b50238125ad14920bf820245406918c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.245200 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 3622008bbcf94ebcb2c2b78c7237b05a (table default.Table_99963fe4c795413c98b34872a5ac9b5f [id=7a192bdbbc3b4571963f35dcd4b589d6]) successfully deleted
I20240506 19:56:34.249918 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 27b24fa4663244a88dac9ea99fe38f3a
I20240506 19:56:34.253363 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3501606fb8de42f68fd7894fa92e1a95" } modify_external_catalogs: true
I20240506 19:56:34.254992 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 7bdcdbb9cf3e4e9db6174bce7785565e (table default.Table_3a86534bf1014d9890190f09491e899d [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:34.255920 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6bc0ddcaf1a2494facb7e66847623940"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.256804 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bc0ddcaf1a2494facb7e66847623940 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.258947 12515 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:34.261008 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_577b201c2bf345bdbaeb7bda56b8296c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.263662 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_577b201c2bf345bdbaeb7bda56b8296c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.273598 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3501606fb8de42f68fd7894fa92e1a95" } modify_external_catalogs: true
I20240506 19:56:34.274395 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 30d819d1237f4798b4847fdf378cbc62
I20240506 19:56:34.280187 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_558320fe838a40a0af9f6e04106029b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:34.281652 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:34.285548 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b63857fa6004468cad193ed4f2a9d366 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]) successfully deleted
I20240506 19:56:34.288318 13095 tablet_service.cc:1467] Processing CreateTablet for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (DEFAULT_TABLE table=default.Table_60bc98ea5883422dac4600f737dc29f1 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.291793 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f91888e88234cbdbd00f48dc1574653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.291744 13244 consensus_queue.cc:260] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.294945 13124 raft_consensus.cc:2454] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:34.295001 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d819d1237f4798b4847fdf378cbc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.295758 12922 leader_election.cc:302] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:34.295903 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_558320fe838a40a0af9f6e04106029b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.296293 13353 raft_consensus.cc:2781] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:34.296404 13353 raft_consensus.cc:482] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:34.296557 13353 raft_consensus.cc:3037] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:34.296805 13089 tablet_service.cc:1467] Processing CreateTablet for tablet b529bb9362924caca399491e7d9e03cd (DEFAULT_TABLE table=default.Table_ddfa1d6244bd47dab51027f3945ed50f [id=fd3051399ebf4aa2b31bbd709b120d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.297314 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d380bf30599d4130bbb80b4f26837779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.297932 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.298657 13242 tablet_replica.cc:330] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:34.299906 12826 tablet_service.cc:1467] Processing CreateTablet for tablet d380bf30599d4130bbb80b4f26837779 (DEFAULT_TABLE table=default.Table_47a20691cc45428695761f4e7ef2e4fc [id=873dfe33ce7141f2bc25d7575a63fc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.300882 13242 raft_consensus.cc:2229] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.301086 13242 raft_consensus.cc:2258] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.301504 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1667a0bbd3af4d678434b068c47cc518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.301729 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 1667a0bbd3af4d678434b068c47cc518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.302104 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1667a0bbd3af4d678434b068c47cc518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.302372 13244 ts_tablet_manager.cc:1398] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.090s user 0.001s sys 0.002s
I20240506 19:56:34.302966 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d380bf30599d4130bbb80b4f26837779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.306241 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b529bb9362924caca399491e7d9e03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.307008 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b529bb9362924caca399491e7d9e03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.310448 13242 ts_tablet_manager.cc:1875] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.311908 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.312487 13244 tablet_bootstrap.cc:492] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:34.315265 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.322149 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 1667a0bbd3af4d678434b068c47cc518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.326026 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5988b89cb52e4f858b52862138bed025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.326128 13252 raft_consensus.cc:482] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.330060 13252 raft_consensus.cc:504] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.331723 13252 leader_election.cc:288] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:34.331988 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d819d1237f4798b4847fdf378cbc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.332418 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_685d0e00c37640ab99bfe74c85c1c0f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.332669 12781 leader_election.cc:341] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 6e60cddf90fe4891be3c60651ebbb184
W20240506 19:56:34.333317 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_685d0e00c37640ab99bfe74c85c1c0f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.335374 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ddfa1d6244bd47dab51027f3945ed50f" } new_table_name: "default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c" modify_external_catalogs: true
I20240506 19:56:34.337174 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bd219422c6e141dea81c43e0227f351c" } new_table_name: "default.Table_62b69a40ec6a432186ff79fc5693659e" modify_external_catalogs: true
I20240506 19:56:34.338727 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3322d5e06d7f437faf751479f7bd019f" } new_table_name: "default.Table_503cbc59b232469d96f78bbec69b3d57" modify_external_catalogs: true
I20240506 19:56:34.340420 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_60bc98ea5883422dac4600f737dc29f1" } new_table_name: "default.Table_e5789654f78a46f79a248c9ed47f314f" modify_external_catalogs: true
I20240506 19:56:34.342866 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c656e16c1045dea79bc23b64fdd5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.345494 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2b36704422c0483db4db7ab48455482d" } new_table_name: "default.Table_ac80e9e0789c4cbaab8826eb3c0bc621" modify_external_catalogs: true
I20240506 19:56:34.346837 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet c51b52a98cec48fc959b01a2a0a8795c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.348904 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f91888e88234cbdbd00f48dc1574653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.354435 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f1e4efbe8b40b199fcb3020569e1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.355072 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e60cddf90fe4891be3c60651ebbb184" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:34.359045 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d57249f30a44dab3384092ec20b515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.359083 12783 leader_election.cc:302] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:34.359664 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e60cddf90fe4891be3c60651ebbb184 with delete_type TABLET_DATA_DELETED (replaced by tablet 5dc0ac37f67942fc8a31d8d43c839246) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.361542 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9d7abbf595704ae18e43dc8996a642aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.364982 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d7abbf595704ae18e43dc8996a642aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.365449 13252 raft_consensus.cc:2726] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:34.370018 13244 tablet_bootstrap.cc:654] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.378880 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.379930 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.400079 13242 ts_tablet_manager.cc:1888] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.421823 13246 ts_tablet_manager.cc:1888] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.422024 13246 log.cc:1196] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/c51b52a98cec48fc959b01a2a0a8795c
I20240506 19:56:34.422435 13246 ts_tablet_manager.cc:1909] T c51b52a98cec48fc959b01a2a0a8795c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.423285 13242 log.cc:1196] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/162a5d3716c343139484a623105825a5
I20240506 19:56:34.423995 13242 ts_tablet_manager.cc:1909] T 162a5d3716c343139484a623105825a5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:34.429495 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 162a5d3716c343139484a623105825a5 (table default.Table_d36494da319a48e18299fbb1708e484b [id=d1e6677412624f19917aae666e3e5184]) successfully deleted
I20240506 19:56:34.432118 13245 tablet_bootstrap.cc:492] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.432356 13245 ts_tablet_manager.cc:1367] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.370s user 0.000s sys 0.008s
I20240506 19:56:34.433413 13245 raft_consensus.cc:348] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.433710 13245 raft_consensus.cc:374] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.433797 13245 raft_consensus.cc:729] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.434183 13245 consensus_queue.cc:260] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:34.435995 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa consensus state has no leader
I20240506 19:56:34.436342 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 44de7153689b4dbd80dae87d3ee48a83 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.437243 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bbc623ea167c48929e77cee2b3d07aaa" } modify_external_catalogs: true
I20240506 19:56:34.437618 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bbc623ea167c48929e77cee2b3d07aaa" } modify_external_catalogs: true
W20240506 19:56:34.438324 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b529bb9362924caca399491e7d9e03cd consensus state has no leader
I20240506 19:56:34.438841 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_df1733520c4a4ae6a99a78f194d6cf06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.439728 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df1733520c4a4ae6a99a78f194d6cf06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.440677 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet a16704567ef6454eaef886fb9cd599ce from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.442224 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 10db9b8cad5045309dff3751cefce838 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:34.443053 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d5513f4926e41a6b78f02a5126cc13a from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:34.443864 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6156c6f582b747de973b357b6d071a46 consensus state has no leader
W20240506 19:56:34.444733 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5988b89cb52e4f858b52862138bed025 consensus state has no leader
W20240506 19:56:34.445463 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 393073bca2a04d15b3e583700262f3e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 393073bca2a04d15b3e583700262f3e0 already in progress: creating tablet
W20240506 19:56:34.445717 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 6d8a9ecf47d7474c82758417449d940c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d8a9ecf47d7474c82758417449d940c already in progress: creating tablet
W20240506 19:56:34.445928 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f66f1e46cce47639fe644fb4ed2e5c0 already in progress: creating tablet
I20240506 19:56:34.447548 13353 raft_consensus.cc:504] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.448648 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:34.448761 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1f1e4efbe8b40b199fcb3020569e1a1 consensus state has no leader
W20240506 19:56:34.449661 12921 leader_election.cc:341] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.450220 13353 leader_election.cc:288] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:34.450373 13353 raft_consensus.cc:482] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.450520 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:34.450528 13353 raft_consensus.cc:504] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.450712 13124 raft_consensus.cc:3037] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:34.451597 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea09efd4aa4019911dd23774f464a2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:34.451824 13123 raft_consensus.cc:2454] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:34.452271 12922 leader_election.cc:302] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:34.453794 13124 raft_consensus.cc:2454] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:34.454070 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea09efd4aa4019911dd23774f464a2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:34.454310 12848 raft_consensus.cc:2454] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:34.454973 12922 leader_election.cc:302] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.456070 13353 leader_election.cc:288] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:34.456277 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1488328703024ae8983add9e4d6afc8f
I20240506 19:56:34.456648 13353 raft_consensus.cc:2781] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:34.456755 13353 raft_consensus.cc:482] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:34.456866 13353 raft_consensus.cc:3037] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:34.458750 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 1488328703024ae8983add9e4d6afc8f (DEFAULT_TABLE table=default.Table_3501606fb8de42f68fd7894fa92e1a95 [id=3cabe6d9486d4b6fa610e1a880a5cdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.459177 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1488328703024ae8983add9e4d6afc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.466363 13092 tablet_service.cc:1467] Processing CreateTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 (DEFAULT_TABLE table=default.Table_92531c4a95d14b50b186416c9cfd94b8 [id=bf75c039ad324e50a2ccb889439e2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.466866 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca394b7f53ce445e8778dd25d0cf25c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.469266 13245 ts_tablet_manager.cc:1398] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20240506 19:56:34.477444 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1488328703024ae8983add9e4d6afc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.480019 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d843fe82af1148f4a9045ce8b09ad11d
I20240506 19:56:34.481571 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:34.482172 12517 catalog_manager.cc:6554] ReplaceTablet: tablet a16704567ef6454eaef886fb9cd599ce deleted and replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:34.482578 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca44a4cb8344307b94c8fafba143836 with delete_type TABLET_DATA_DELETED (replaced by tablet 92951843c8bd446aa86436b65d9eb793) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.483592 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet d843fe82af1148f4a9045ce8b09ad11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.485179 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet a16704567ef6454eaef886fb9cd599ce with delete_type TABLET_DATA_DELETED (replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.487383 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b0e5bbcbcb324f54bc4feab7c750c029"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.487969 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0e5bbcbcb324f54bc4feab7c750c029 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.487931 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_93498ac3d84b4de4af0e703aa2b03878"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.488409 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93498ac3d84b4de4af0e703aa2b03878 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.490345 13246 tablet_replica.cc:330] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.490546 13246 raft_consensus.cc:2229] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:34.490784 13246 pending_rounds.cc:62] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:34.490847 13246 pending_rounds.cc:68] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744013231804416 op_type: NO_OP noop_request { }
I20240506 19:56:34.490978 13246 raft_consensus.cc:2866] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:34.491027 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet a16704567ef6454eaef886fb9cd599ce with delete_type TABLET_DATA_DELETED (replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.491067 13246 raft_consensus.cc:2258] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.491762 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 1667a0bbd3af4d678434b068c47cc518 (DEFAULT_TABLE table=default.Table_dfd40b6029db498e870d3e8a44ae6bdd [id=c4a20cff71d7496e8b07f477a0c1ea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.491919 13246 ts_tablet_manager.cc:1875] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.492480 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet d843fe82af1148f4a9045ce8b09ad11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.492380 13311 raft_consensus.cc:504] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.494833 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet c51b52a98cec48fc959b01a2a0a8795c (table default.Table_2a4c70f40666495ba552107168859093 [id=513f5de1c20e4aa48bd9f216c869b5fe]) successfully deleted
I20240506 19:56:34.495311 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 1488328703024ae8983add9e4d6afc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.496595 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1667a0bbd3af4d678434b068c47cc518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.500620 13246 ts_tablet_manager.cc:1888] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:34.500887 13246 log.cc:1196] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/6e60cddf90fe4891be3c60651ebbb184
I20240506 19:56:34.501410 13246 ts_tablet_manager.cc:1909] T 6e60cddf90fe4891be3c60651ebbb184 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.501420 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4e6c606a634df58f2180361078d3d5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:34.502177 12921 leader_election.cc:341] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.502470 13249 tablet_bootstrap.cc:492] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:34.502638 13249 ts_tablet_manager.cc:1367] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.330s user 0.050s sys 0.153s
I20240506 19:56:34.502916 13311 leader_election.cc:288] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:34.503085 13311 raft_consensus.cc:2781] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:34.503265 13311 raft_consensus.cc:686] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:34.503598 13311 consensus_queue.cc:237] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.505739 13249 raft_consensus.cc:348] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.505972 13249 raft_consensus.cc:374] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.506072 13249 raft_consensus.cc:729] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.506289 13249 consensus_queue.cc:260] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.506893 13249 ts_tablet_manager.cc:1398] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:34.508387 13249 tablet_bootstrap.cc:492] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:34.513537 13245 tablet_bootstrap.cc:492] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.516135 13245 tablet_bootstrap.cc:654] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.519951 13249 tablet_bootstrap.cc:654] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.520422 13246 tablet_replica.cc:330] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.520640 13246 raft_consensus.cc:2229] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:34.520932 13246 pending_rounds.cc:62] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:34.521021 13246 pending_rounds.cc:68] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744014137372672 op_type: NO_OP noop_request { }
I20240506 19:56:34.521219 13246 raft_consensus.cc:2866] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:34.525360 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d819d1237f4798b4847fdf378cbc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.525521 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 162a5d3716c343139484a623105825a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.525642 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 1667a0bbd3af4d678434b068c47cc518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.526006 12962 tablet_service.cc:1467] Processing CreateTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 (DEFAULT_TABLE table=default.Table_92531c4a95d14b50b186416c9cfd94b8 [id=bf75c039ad324e50a2ccb889439e2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.526455 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca394b7f53ce445e8778dd25d0cf25c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.526399 12960 tablet_service.cc:1467] Processing CreateTablet for tablet b529bb9362924caca399491e7d9e03cd (DEFAULT_TABLE table=default.Table_ddfa1d6244bd47dab51027f3945ed50f [id=fd3051399ebf4aa2b31bbd709b120d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.526695 13246 raft_consensus.cc:2258] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.526713 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 1488328703024ae8983add9e4d6afc8f (DEFAULT_TABLE table=default.Table_3501606fb8de42f68fd7894fa92e1a95 [id=3cabe6d9486d4b6fa610e1a880a5cdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.526813 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b529bb9362924caca399491e7d9e03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.526887 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet a16704567ef6454eaef886fb9cd599ce with delete_type TABLET_DATA_DELETED (replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.527115 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 1488328703024ae8983add9e4d6afc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.527047 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet d843fe82af1148f4a9045ce8b09ad11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.527302 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1488328703024ae8983add9e4d6afc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.527896 13246 ts_tablet_manager.cc:1875] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.530277 13245 tablet_bootstrap.cc:492] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.530426 13245 ts_tablet_manager.cc:1367] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
W20240506 19:56:34.530788 13313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:34.531287 13313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20240506 19:56:34.531215 13245 raft_consensus.cc:348] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:34.531441 13313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240506 19:56:34.531499 13245 raft_consensus.cc:374] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.531605 13245 raft_consensus.cc:729] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.531911 13245 consensus_queue.cc:260] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.532657 13245 ts_tablet_manager.cc:1398] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240506 19:56:34.534152 13245 tablet_bootstrap.cc:492] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.535137 13329 raft_consensus.cc:482] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.535276 13329 raft_consensus.cc:504] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.535959 13329 leader_election.cc:288] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:34.536021 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.539117 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2abecd9ec7e4606b38c7d22477ddf65" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:34.539870 13058 leader_election.cc:341] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:34.543668 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dae4e46e818c4147960f5d606ce3b969" } modify_external_catalogs: true
I20240506 19:56:34.543891 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dae4e46e818c4147960f5d606ce3b969" } modify_external_catalogs: true
I20240506 19:56:34.544239 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 6e60cddf90fe4891be3c60651ebbb184 (table default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:34.544411 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_675510c53e1848ed8176a861a1ace8c4" } modify_external_catalogs: true
I20240506 19:56:34.544600 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_675510c53e1848ed8176a861a1ace8c4" } modify_external_catalogs: true
W20240506 19:56:34.544631 13313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 19:56:34.544744 13313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:34.544811 13313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 19:56:34.545071 13313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 19:56:34.544412 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_038664bd22da4e20b40b21843516298f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.545519 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_038664bd22da4e20b40b21843516298f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.545627 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 1488328703024ae8983add9e4d6afc8f (DEFAULT_TABLE table=default.Table_3501606fb8de42f68fd7894fa92e1a95 [id=3cabe6d9486d4b6fa610e1a880a5cdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.546072 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1488328703024ae8983add9e4d6afc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.550812 12826 tablet_service.cc:1467] Processing CreateTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 (DEFAULT_TABLE table=default.Table_92531c4a95d14b50b186416c9cfd94b8 [id=bf75c039ad324e50a2ccb889439e2b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.551415 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca394b7f53ce445e8778dd25d0cf25c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.552708 13246 ts_tablet_manager.cc:1888] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:34.552891 13246 log.cc:1196] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/7bdcdbb9cf3e4e9db6174bce7785565e
I20240506 19:56:34.557075 13246 ts_tablet_manager.cc:1909] T 7bdcdbb9cf3e4e9db6174bce7785565e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.559377 13246 tablet_replica.cc:330] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.553288 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7b7dd900dc5b478280237dfd98e0f303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:34.560729 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b7dd900dc5b478280237dfd98e0f303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:34.561036 13246 raft_consensus.cc:2229] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.561139 13242 tablet_replica.cc:330] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:34.561326 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.561398 13242 raft_consensus.cc:2229] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.561597 13242 raft_consensus.cc:2258] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:34.561064 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 27b24fa4663244a88dac9ea99fe38f3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b24fa4663244a88dac9ea99fe38f3a already in progress: creating tablet
W20240506 19:56:34.563144 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1667a0bbd3af4d678434b068c47cc518 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1667a0bbd3af4d678434b068c47cc518 already in progress: creating tablet
W20240506 19:56:34.563484 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 30d819d1237f4798b4847fdf378cbc62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30d819d1237f4798b4847fdf378cbc62 already in progress: creating tablet
I20240506 19:56:34.561517 13246 raft_consensus.cc:2258] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.564132 13313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.253:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.48.253
--webserver_port=35887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:34.564383 13246 ts_tablet_manager.cc:1875] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.564972 13242 ts_tablet_manager.cc:1875] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.560005 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c" } modify_external_catalogs: true
I20240506 19:56:34.565960 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c" } modify_external_catalogs: true
I20240506 19:56:34.569509 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 7bdcdbb9cf3e4e9db6174bce7785565e (table default.Table_3a86534bf1014d9890190f09491e899d [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:34.569603 13313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20240506 19:56:34.569846 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d0e3260f46024386baac64cb8677883c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0e3260f46024386baac64cb8677883c already in progress: creating tablet
I20240506 19:56:34.572641 13313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 19:56:34.576184 13246 ts_tablet_manager.cc:1888] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.576833 13246 log.cc:1196] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1d5513f4926e41a6b78f02a5126cc13a
I20240506 19:56:34.577179 13242 ts_tablet_manager.cc:1888] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.577484 13246 ts_tablet_manager.cc:1909] T 1d5513f4926e41a6b78f02a5126cc13a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.578644 13353 raft_consensus.cc:504] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.582134 13246 tablet_replica.cc:330] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.582587 13246 raft_consensus.cc:2229] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.583082 13246 raft_consensus.cc:2258] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.583487 13353 leader_election.cc:288] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:34.582998 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea09efd4aa4019911dd23774f464a2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:34.584021 12848 raft_consensus.cc:3037] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:34.585640 13246 ts_tablet_manager.cc:1875] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.586853 12848 raft_consensus.cc:2454] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:34.587530 12921 leader_election.cc:302] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:34.587841 13353 raft_consensus.cc:2781] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:34.587991 13242 log.cc:1196] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/6e60cddf90fe4891be3c60651ebbb184
I20240506 19:56:34.588083 13353 raft_consensus.cc:686] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:34.588451 13242 ts_tablet_manager.cc:1909] T 6e60cddf90fe4891be3c60651ebbb184 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:34.588451 13353 consensus_queue.cc:237] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:34.589457 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 6e60cddf90fe4891be3c60651ebbb184 (table default.Table_5c4383f23b354870b6fddac21d4a453a [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
W20240506 19:56:34.589787 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a16704567ef6454eaef886fb9cd599ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a16704567ef6454eaef886fb9cd599ce already in progress: creating tablet
W20240506 19:56:34.590047 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d843fe82af1148f4a9045ce8b09ad11d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d843fe82af1148f4a9045ce8b09ad11d already in progress: creating tablet
W20240506 19:56:34.590314 12507 catalog_manager.cc:4797] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1488328703024ae8983add9e4d6afc8f because the tablet was not found. No further retry: Not found: Tablet not found: 1488328703024ae8983add9e4d6afc8f
I20240506 19:56:34.595328 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_86cbff0af1184af6a66af4ee25521f9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:34.596803 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 2502b423bd1040539fb1a3ebbece3c1c from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:34.598114 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 143159us
I20240506 19:56:34.599401 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4e6c606a634df58f2180361078d3d5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:34.600155 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 89eb4e2701894f32a85d2cd89da58ec2
I20240506 19:56:34.600943 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 903ce1bc6ae7488d83bb0945424ed9f9
I20240506 19:56:34.601043 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b529bb9362924caca399491e7d9e03cd
I20240506 19:56:34.601230 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea09efd4aa4019911dd23774f464a2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:34.601500 13124 raft_consensus.cc:3037] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:34.601974 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet b529bb9362924caca399491e7d9e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.602663 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:34.603032 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b529bb9362924caca399491e7d9e03cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b529bb9362924caca399491e7d9e03cd already in progress: creating tablet
W20240506 19:56:34.603406 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 903ce1bc6ae7488d83bb0945424ed9f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903ce1bc6ae7488d83bb0945424ed9f9 already in progress: creating tablet
I20240506 19:56:34.603637 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.605060 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet b529bb9362924caca399491e7d9e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.605501 13124 raft_consensus.cc:2454] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:34.606777 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.608229 13244 tablet_bootstrap.cc:492] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:34.608562 13246 ts_tablet_manager.cc:1888] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.609414 13246 log.cc:1196] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ab4e6c606a634df58f2180361078d3d5
I20240506 19:56:34.610080 13246 ts_tablet_manager.cc:1909] T ab4e6c606a634df58f2180361078d3d5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.611831 13245 tablet_bootstrap.cc:654] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.611843 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 0114edac976844619e438c17bb959010 (DEFAULT_TABLE table=default.Table_1ba0602f38e64789a126e75fc115f5c2 [id=c2803598df8241beb0b6ce684515661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.612437 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0114edac976844619e438c17bb959010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.612938 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1d5513f4926e41a6b78f02a5126cc13a (table default.Table_209a7bac6fc84e80afea650dd487e1e0 [id=1ef16dfbd249466fa6b0a7df43515b7b]) successfully deleted
W20240506 19:56:34.613345 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a80834187f874661b611b43065eee943 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a80834187f874661b611b43065eee943 already in progress: creating tablet
I20240506 19:56:34.613701 13244 ts_tablet_manager.cc:1367] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.311s user 0.012s sys 0.001s
W20240506 19:56:34.614655 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86cbff0af1184af6a66af4ee25521f9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:34.614737 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8842523a97954e3f81ffd87530201d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8842523a97954e3f81ffd87530201d93 already in progress: creating tablet
W20240506 19:56:34.615172 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 79486583025c42bfb1321f9cb6dd96f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79486583025c42bfb1321f9cb6dd96f1 already in progress: creating tablet
W20240506 19:56:34.615449 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 18920e08b0de4f5292abb54e3432ee39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18920e08b0de4f5292abb54e3432ee39 already in progress: creating tablet
I20240506 19:56:34.616667 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.617358 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2502b423bd1040539fb1a3ebbece3c1c
I20240506 19:56:34.617434 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2abecd9ec7e4606b38c7d22477ddf65" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:34.618044 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 2502b423bd1040539fb1a3ebbece3c1c deleted and replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14
W20240506 19:56:34.618292 13057 leader_election.cc:341] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:34.618367 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c with delete_type TABLET_DATA_DELETED (replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.618476 13057 leader_election.cc:302] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.619002 13329 raft_consensus.cc:2726] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:34.619936 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c with delete_type TABLET_DATA_DELETED (replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.620535 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 0114edac976844619e438c17bb959010 (DEFAULT_TABLE table=default.Table_1ba0602f38e64789a126e75fc115f5c2 [id=c2803598df8241beb0b6ce684515661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.621117 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0114edac976844619e438c17bb959010. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:34.622651 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2502b423bd1040539fb1a3ebbece3c1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2502b423bd1040539fb1a3ebbece3c1c already in progress: creating tablet
I20240506 19:56:34.624325 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.624475 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (DEFAULT_TABLE table=default.Table_9d7abbf595704ae18e43dc8996a642aa [id=f13c2a81c2384d5bb7dbe0cef8718f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.624931 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41faccd54f3e49ddb48ea5234f6cdab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.625645 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (DEFAULT_TABLE table=default.Table_9d7abbf595704ae18e43dc8996a642aa [id=f13c2a81c2384d5bb7dbe0cef8718f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.626055 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41faccd54f3e49ddb48ea5234f6cdab5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:34.629683 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 89eb4e2701894f32a85d2cd89da58ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89eb4e2701894f32a85d2cd89da58ec2 already in progress: creating tablet
I20240506 19:56:34.629920 13246 tablet_replica.cc:330] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.630422 13246 raft_consensus.cc:2229] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.630182 13244 raft_consensus.cc:348] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.630542 13244 raft_consensus.cc:374] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.630656 13244 raft_consensus.cc:729] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.631062 13246 raft_consensus.cc:2258] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.630941 13244 consensus_queue.cc:260] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.631286 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c (DEFAULT_TABLE table=default.Table_b50238125ad14920bf820245406918c3 [id=7f61579ec04443a6992e572f3e52f980]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:34.631441 12922 leader_election.cc:341] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:34.631706 12922 leader_election.cc:302] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.631803 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618c56ef126a4ca48b6a379a6771bc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.632316 13246 ts_tablet_manager.cc:1875] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.632927 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c (DEFAULT_TABLE table=default.Table_b50238125ad14920bf820245406918c3 [id=7f61579ec04443a6992e572f3e52f980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.633277 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618c56ef126a4ca48b6a379a6771bc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.633764 12962 tablet_service.cc:1467] Processing CreateTablet for tablet ea25fb8e42234b82885b8fb2275d4925 (DEFAULT_TABLE table=default.Table_685d0e00c37640ab99bfe74c85c1c0f3 [id=8b3a851d7ee4438daf7440e2af8f9885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.634153 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea25fb8e42234b82885b8fb2275d4925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.634860 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 (DEFAULT_TABLE table=default.Table_577b201c2bf345bdbaeb7bda56b8296c [id=0772215de50d467585bec1f234187f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.635250 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2005ef58fd4fcea16abe771ee385f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.635955 12957 tablet_service.cc:1467] Processing CreateTablet for tablet a36b4d51b100486c831a41a52d81fd66 (DEFAULT_TABLE table=default.Table_e5d18039b7f04edaa0c985be5260be35 [id=7a6f6d0d394340ea93a13720ea7b769e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.636365 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36b4d51b100486c831a41a52d81fd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.636962 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 07755e37a20646a3b1a7b900d54a077c (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.637346 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07755e37a20646a3b1a7b900d54a077c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.647220 12819 tablet_service.cc:1467] Processing CreateTablet for tablet a36b4d51b100486c831a41a52d81fd66 (DEFAULT_TABLE table=default.Table_e5d18039b7f04edaa0c985be5260be35 [id=7a6f6d0d394340ea93a13720ea7b769e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.647738 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36b4d51b100486c831a41a52d81fd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.648079 13246 ts_tablet_manager.cc:1888] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.651453 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 07755e37a20646a3b1a7b900d54a077c (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.651897 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07755e37a20646a3b1a7b900d54a077c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.655781 13246 log.cc:1196] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/0ca44a4cb8344307b94c8fafba143836
I20240506 19:56:34.656441 13246 ts_tablet_manager.cc:1909] T 0ca44a4cb8344307b94c8fafba143836 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.657469 12816 tablet_service.cc:1467] Processing CreateTablet for tablet ea25fb8e42234b82885b8fb2275d4925 (DEFAULT_TABLE table=default.Table_685d0e00c37640ab99bfe74c85c1c0f3 [id=8b3a851d7ee4438daf7440e2af8f9885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.657930 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea25fb8e42234b82885b8fb2275d4925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.659840 13246 tablet_replica.cc:330] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.660310 13246 raft_consensus.cc:2229] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.660872 13246 raft_consensus.cc:2258] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.661995 13246 ts_tablet_manager.cc:1875] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.662959 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa (DEFAULT_TABLE table=default.Table_558320fe838a40a0af9f6e04106029b0 [id=5d61a15249c74e509342122448b76500]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.663472 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2519c03738864f6ab5dcfab7bc3fc8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.665660 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 (DEFAULT_TABLE table=default.Table_577b201c2bf345bdbaeb7bda56b8296c [id=0772215de50d467585bec1f234187f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.666282 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2005ef58fd4fcea16abe771ee385f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.676014 13246 ts_tablet_manager.cc:1888] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.676553 13246 log.cc:1196] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d2abecd9ec7e4606b38c7d22477ddf65
I20240506 19:56:34.677158 13246 ts_tablet_manager.cc:1909] T d2abecd9ec7e4606b38c7d22477ddf65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.712383 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa (DEFAULT_TABLE table=default.Table_558320fe838a40a0af9f6e04106029b0 [id=5d61a15249c74e509342122448b76500]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.712910 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2519c03738864f6ab5dcfab7bc3fc8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.720005 13243 ts_tablet_manager.cc:1888] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.720178 13243 log.cc:1196] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d2abecd9ec7e4606b38c7d22477ddf65
I20240506 19:56:34.720644 13243 ts_tablet_manager.cc:1909] T d2abecd9ec7e4606b38c7d22477ddf65 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:34.721804 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 90352.1us
I20240506 19:56:34.722931 13311 raft_consensus.cc:2726] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:34.724637 12956 tablet_service.cc:1467] Processing CreateTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad (DEFAULT_TABLE table=default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc [id=49660c755f8f470b8b8dc301ca27bb26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.724795 12955 tablet_service.cc:1467] Processing CreateTablet for tablet aae313961a914ed485f866d85785a788 (DEFAULT_TABLE table=default.Table_86cbff0af1184af6a66af4ee25521f9d [id=a63e1c9e058a4501831a67516affe578]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.725265 12959 tablet_service.cc:1467] Processing CreateTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 (DEFAULT_TABLE table=default.Table_038664bd22da4e20b40b21843516298f [id=34aeae2e9b5546d1b405ce1432e3c0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.725374 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae313961a914ed485f866d85785a788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.729621 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a8b12f36e74c95ad6eaf30e432f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.730341 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e7329b22414da886dee4e1ec6bf3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.731495 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.731848 13246 tablet_replica.cc:330] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:34.732064 13246 raft_consensus.cc:2229] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.732250 13246 raft_consensus.cc:2258] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.732829 13246 ts_tablet_manager.cc:1875] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.734809 12957 tablet_service.cc:1467] Processing CreateTablet for tablet d3ec577dd06640068575288c24e1105a (DEFAULT_TABLE table=default.Table_b0e5bbcbcb324f54bc4feab7c750c029 [id=436b5c4427c24f4caf61a5c06a923b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.735733 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f (DEFAULT_TABLE table=default.Table_3a86534bf1014d9890190f09491e899d [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.739396 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet b529bb9362924caca399491e7d9e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.739496 12958 tablet_service.cc:1467] Processing CreateTablet for tablet b5a443c0d35741b9966b32a7808a97e2 (DEFAULT_TABLE table=default.Table_93498ac3d84b4de4af0e703aa2b03878 [id=54286945a7554349b48aa6511f84326c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.739850 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a443c0d35741b9966b32a7808a97e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.745018 13246 ts_tablet_manager.cc:1888] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.745147 13246 log.cc:1196] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/903ce1bc6ae7488d83bb0945424ed9f9
I20240506 19:56:34.745651 13244 ts_tablet_manager.cc:1398] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.132s user 0.001s sys 0.001s
I20240506 19:56:34.746846 12963 tablet_service.cc:1467] Processing CreateTablet for tablet cc8335f9308046279a25a4a95d8e0f65 (DEFAULT_TABLE table=default.Table_df1733520c4a4ae6a99a78f194d6cf06 [id=64a093df5c7143668da7d3e91eccba2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.747216 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8335f9308046279a25a4a95d8e0f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.749621 12961 tablet_service.cc:1467] Processing CreateTablet for tablet d05c5ca7ff864fa98f3494032729e978 (DEFAULT_TABLE table=default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.749678 13244 tablet_bootstrap.cc:492] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:34.750000 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05c5ca7ff864fa98f3494032729e978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.753592 13244 tablet_bootstrap.cc:654] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.753919 13246 ts_tablet_manager.cc:1909] T 903ce1bc6ae7488d83bb0945424ed9f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:34.759245 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c with delete_type TABLET_DATA_DELETED (replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:34.760742 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 10db9b8cad5045309dff3751cefce838 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:34.760704 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 0114edac976844619e438c17bb959010 (DEFAULT_TABLE table=default.Table_1ba0602f38e64789a126e75fc115f5c2 [id=c2803598df8241beb0b6ce684515661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.760916 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 (DEFAULT_TABLE table=default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.761377 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00de09ddcdd948a6a3500a6c9d814c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.761535 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0114edac976844619e438c17bb959010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.766146 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ec577dd06640068575288c24e1105a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.766427 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c1590e36b4471d8e72f4b07410ee0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.776363 13353 raft_consensus.cc:482] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.776547 13353 raft_consensus.cc:504] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
W20240506 19:56:34.778270 12921 leader_election.cc:341] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Not found: Tablet not found: 162a5d3716c343139484a623105825a5
I20240506 19:56:34.778489 13353 leader_election.cc:288] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:34.779837 12922 leader_election.cc:341] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 162a5d3716c343139484a623105825a5
I20240506 19:56:34.780925 12817 tablet_service.cc:1467] Processing CreateTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad (DEFAULT_TABLE table=default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc [id=49660c755f8f470b8b8dc301ca27bb26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.781823 12820 tablet_service.cc:1467] Processing CreateTablet for tablet cc8335f9308046279a25a4a95d8e0f65 (DEFAULT_TABLE table=default.Table_df1733520c4a4ae6a99a78f194d6cf06 [id=64a093df5c7143668da7d3e91eccba2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.782642 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8335f9308046279a25a4a95d8e0f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.785913 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 (DEFAULT_TABLE table=default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.786268 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00de09ddcdd948a6a3500a6c9d814c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.785737 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f (DEFAULT_TABLE table=default.Table_3a86534bf1014d9890190f09491e899d [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.793175 12821 tablet_service.cc:1467] Processing CreateTablet for tablet d3ec577dd06640068575288c24e1105a (DEFAULT_TABLE table=default.Table_b0e5bbcbcb324f54bc4feab7c750c029 [id=436b5c4427c24f4caf61a5c06a923b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.793874 12827 tablet_service.cc:1467] Processing CreateTablet for tablet b5a443c0d35741b9966b32a7808a97e2 (DEFAULT_TABLE table=default.Table_93498ac3d84b4de4af0e703aa2b03878 [id=54286945a7554349b48aa6511f84326c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.794365 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e7329b22414da886dee4e1ec6bf3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.797556 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c1590e36b4471d8e72f4b07410ee0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.801455 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a443c0d35741b9966b32a7808a97e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.805043 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ec577dd06640068575288c24e1105a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.830220 12819 tablet_service.cc:1467] Processing CreateTablet for tablet d05c5ca7ff864fa98f3494032729e978 (DEFAULT_TABLE table=default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.830999 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05c5ca7ff864fa98f3494032729e978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.834446 13249 tablet_bootstrap.cc:492] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:34.837319 13249 ts_tablet_manager.cc:1367] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.329s user 0.051s sys 0.053s
I20240506 19:56:34.838486 13249 raft_consensus.cc:348] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.838809 13249 raft_consensus.cc:374] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.838893 13249 raft_consensus.cc:729] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.839164 13249 consensus_queue.cc:260] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:34.839468 13381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:34.839707 13249 ts_tablet_manager.cc:1398] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:34.841914 13249 tablet_bootstrap.cc:492] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:34.846537 13249 tablet_bootstrap.cc:654] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:34.851119 13379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:34.851591 12922 leader_election.cc:302] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:34.852113 13353 raft_consensus.cc:2726] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:34.852174 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 72338.9us
I20240506 19:56:34.854539 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d2abecd9ec7e4606b38c7d22477ddf65 (table default.Table_eb361931fc51481590136c98bc041bc6 [id=031d0b422342478ba95396f085780f3b]) successfully deleted
I20240506 19:56:34.854964 13243 tablet_replica.cc:330] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:34.855131 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8842523a97954e3f81ffd87530201d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8842523a97954e3f81ffd87530201d93 already in progress: creating tablet
I20240506 19:56:34.855283 13243 raft_consensus.cc:2229] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:34.855378 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 79486583025c42bfb1321f9cb6dd96f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79486583025c42bfb1321f9cb6dd96f1 already in progress: creating tablet
I20240506 19:56:34.855500 13243 raft_consensus.cc:2258] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.856755 13249 tablet_bootstrap.cc:492] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:34.856946 13249 ts_tablet_manager.cc:1367] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240506 19:56:34.858242 13249 raft_consensus.cc:348] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.858547 13249 raft_consensus.cc:374] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.858669 13249 raft_consensus.cc:729] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.859056 13249 consensus_queue.cc:260] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.860050 13243 ts_tablet_manager.cc:1875] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.864544 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (DEFAULT_TABLE table=default.Table_9d7abbf595704ae18e43dc8996a642aa [id=f13c2a81c2384d5bb7dbe0cef8718f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.864703 13245 tablet_bootstrap.cc:492] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.864873 13245 ts_tablet_manager.cc:1367] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.331s user 0.055s sys 0.063s
I20240506 19:56:34.864841 13252 raft_consensus.cc:482] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:34.865036 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41faccd54f3e49ddb48ea5234f6cdab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.865085 13252 raft_consensus.cc:504] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.865711 13245 raft_consensus.cc:348] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.865952 13245 raft_consensus.cc:374] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.866026 13245 raft_consensus.cc:729] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.866003 13252 leader_election.cc:288] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:34.866284 13245 consensus_queue.cc:260] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.867658 13245 ts_tablet_manager.cc:1398] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:34.867990 13245 tablet_bootstrap.cc:492] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.869695 13245 tablet_bootstrap.cc:654] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.872640 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903ce1bc6ae7488d83bb0945424ed9f9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:34.873621 12783 leader_election.cc:341] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 903ce1bc6ae7488d83bb0945424ed9f9
I20240506 19:56:34.873831 12783 leader_election.cc:302] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:34.878700 13243 ts_tablet_manager.cc:1888] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.878849 13243 log.cc:1196] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/18920e08b0de4f5292abb54e3432ee39
I20240506 19:56:34.879031 13252 raft_consensus.cc:2726] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:34.879191 13243 ts_tablet_manager.cc:1909] T 18920e08b0de4f5292abb54e3432ee39 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.880302 13243 tablet_replica.cc:330] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:34.880476 13243 raft_consensus.cc:2229] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.880652 13243 raft_consensus.cc:2258] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.881275 13243 ts_tablet_manager.cc:1875] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.881830 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 18920e08b0de4f5292abb54e3432ee39 (table default.Table_fcd8a4763edf472692c23bb235cb7852 [id=229d13dce55e4738973b0180ddf23c5e]) successfully deleted
I20240506 19:56:34.884709 13245 tablet_bootstrap.cc:492] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.884871 13245 ts_tablet_manager.cc:1367] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20240506 19:56:34.885761 13245 raft_consensus.cc:348] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.885996 13245 raft_consensus.cc:374] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.886101 13245 raft_consensus.cc:729] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.886346 13245 consensus_queue.cc:260] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.887046 13245 ts_tablet_manager.cc:1398] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:34.887094 13098 tablet_service.cc:1467] Processing CreateTablet for tablet ea25fb8e42234b82885b8fb2275d4925 (DEFAULT_TABLE table=default.Table_685d0e00c37640ab99bfe74c85c1c0f3 [id=8b3a851d7ee4438daf7440e2af8f9885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.887349 13245 tablet_bootstrap.cc:492] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.887486 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea25fb8e42234b82885b8fb2275d4925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.889461 13245 tablet_bootstrap.cc:654] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.892457 13243 ts_tablet_manager.cc:1888] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.892583 13243 log.cc:1196] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ab4e6c606a634df58f2180361078d3d5
I20240506 19:56:34.892928 13243 ts_tablet_manager.cc:1909] T ab4e6c606a634df58f2180361078d3d5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.894222 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ab4e6c606a634df58f2180361078d3d5 (table default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]) successfully deleted
I20240506 19:56:34.894721 13243 tablet_replica.cc:330] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:34.894870 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 6d8a9ecf47d7474c82758417449d940c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d8a9ecf47d7474c82758417449d940c already in progress: creating tablet
I20240506 19:56:34.894929 13243 raft_consensus.cc:2229] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.895093 13243 raft_consensus.cc:2258] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:34.895103 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f66f1e46cce47639fe644fb4ed2e5c0 already in progress: creating tablet
W20240506 19:56:34.895329 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 30d819d1237f4798b4847fdf378cbc62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30d819d1237f4798b4847fdf378cbc62 already in progress: creating tablet
I20240506 19:56:34.895660 13243 ts_tablet_manager.cc:1875] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.901427 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c (DEFAULT_TABLE table=default.Table_b50238125ad14920bf820245406918c3 [id=7f61579ec04443a6992e572f3e52f980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.902143 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618c56ef126a4ca48b6a379a6771bc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.903702 13243 ts_tablet_manager.cc:1888] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.903842 13243 log.cc:1196] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/162a5d3716c343139484a623105825a5
I20240506 19:56:34.904170 13243 ts_tablet_manager.cc:1909] T 162a5d3716c343139484a623105825a5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.905223 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 162a5d3716c343139484a623105825a5 (table default.Table_d36494da319a48e18299fbb1708e484b [id=d1e6677412624f19917aae666e3e5184]) successfully deleted
W20240506 19:56:34.905889 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1667a0bbd3af4d678434b068c47cc518 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1667a0bbd3af4d678434b068c47cc518 already in progress: creating tablet
W20240506 19:56:34.906141 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a16704567ef6454eaef886fb9cd599ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a16704567ef6454eaef886fb9cd599ce already in progress: creating tablet
I20240506 19:56:34.906296 13243 tablet_replica.cc:330] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:34.906404 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d843fe82af1148f4a9045ce8b09ad11d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d843fe82af1148f4a9045ce8b09ad11d already in progress: creating tablet
I20240506 19:56:34.906545 13243 raft_consensus.cc:2229] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:34.906821 13243 pending_rounds.cc:62] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:34.906895 13243 pending_rounds.cc:68] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744015889727488 op_type: NO_OP noop_request { }
I20240506 19:56:34.907074 13243 raft_consensus.cc:2866] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240506 19:56:34.907146 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1488328703024ae8983add9e4d6afc8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1488328703024ae8983add9e4d6afc8f already in progress: creating tablet
I20240506 19:56:34.907184 13243 raft_consensus.cc:2258] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:34.907371 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 27b24fa4663244a88dac9ea99fe38f3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b24fa4663244a88dac9ea99fe38f3a already in progress: creating tablet
W20240506 19:56:34.907583 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 89eb4e2701894f32a85d2cd89da58ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89eb4e2701894f32a85d2cd89da58ec2 already in progress: creating tablet
W20240506 19:56:34.907804 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b529bb9362924caca399491e7d9e03cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b529bb9362924caca399491e7d9e03cd already in progress: creating tablet
I20240506 19:56:34.908162 13243 ts_tablet_manager.cc:1875] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.908963 13089 tablet_service.cc:1467] Processing CreateTablet for tablet a36b4d51b100486c831a41a52d81fd66 (DEFAULT_TABLE table=default.Table_e5d18039b7f04edaa0c985be5260be35 [id=7a6f6d0d394340ea93a13720ea7b769e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.909555 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36b4d51b100486c831a41a52d81fd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.911731 13245 tablet_bootstrap.cc:492] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.911886 13245 ts_tablet_manager.cc:1367] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.020s
I20240506 19:56:34.912674 13245 raft_consensus.cc:348] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.912891 13245 raft_consensus.cc:374] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.912973 13245 raft_consensus.cc:729] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.913265 13245 consensus_queue.cc:260] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.914183 13245 ts_tablet_manager.cc:1398] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:34.914554 13245 tablet_bootstrap.cc:492] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.916858 13245 tablet_bootstrap.cc:654] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.919992 13243 ts_tablet_manager.cc:1888] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:34.920189 13243 log.cc:1196] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/903ce1bc6ae7488d83bb0945424ed9f9
I20240506 19:56:34.920677 13243 ts_tablet_manager.cc:1909] T 903ce1bc6ae7488d83bb0945424ed9f9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.922169 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 903ce1bc6ae7488d83bb0945424ed9f9 (table default.Table_675510c53e1848ed8176a861a1ace8c4 [id=2b7d2dfedf4d4169a2244e104dcd154d]) successfully deleted
I20240506 19:56:34.922261 13243 tablet_replica.cc:330] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:34.922173 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 (DEFAULT_TABLE table=default.Table_577b201c2bf345bdbaeb7bda56b8296c [id=0772215de50d467585bec1f234187f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.922447 13243 raft_consensus.cc:2229] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.922631 13243 raft_consensus.cc:2258] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:34.922680 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2502b423bd1040539fb1a3ebbece3c1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2502b423bd1040539fb1a3ebbece3c1c already in progress: creating tablet
I20240506 19:56:34.922701 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2005ef58fd4fcea16abe771ee385f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.923564 13243 ts_tablet_manager.cc:1875] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:34.930662 13243 ts_tablet_manager.cc:1888] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:34.930835 13243 log.cc:1196] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/10db9b8cad5045309dff3751cefce838
I20240506 19:56:34.930852 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa (DEFAULT_TABLE table=default.Table_558320fe838a40a0af9f6e04106029b0 [id=5d61a15249c74e509342122448b76500]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.931031 13101 tablet_service.cc:1467] Processing CreateTablet for tablet aae313961a914ed485f866d85785a788 (DEFAULT_TABLE table=default.Table_86cbff0af1184af6a66af4ee25521f9d [id=a63e1c9e058a4501831a67516affe578]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.931315 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2519c03738864f6ab5dcfab7bc3fc8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.931529 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae313961a914ed485f866d85785a788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.931452 13243 ts_tablet_manager.cc:1909] T 10db9b8cad5045309dff3751cefce838 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:34.933115 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 10db9b8cad5045309dff3751cefce838 (table default.Table_06ec673eaab44720a193e5236fe7481d [id=5f25a2a3fe494e7f88d31b78f2e482c2]) successfully deleted
I20240506 19:56:34.938243 13245 tablet_bootstrap.cc:492] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.938397 13245 ts_tablet_manager.cc:1367] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.011s
I20240506 19:56:34.938730 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 07755e37a20646a3b1a7b900d54a077c (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.939201 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07755e37a20646a3b1a7b900d54a077c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.939424 13245 raft_consensus.cc:348] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.939656 13245 raft_consensus.cc:374] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.939779 13245 raft_consensus.cc:729] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.940140 13245 consensus_queue.cc:260] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.942060 13245 ts_tablet_manager.cc:1398] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240506 19:56:34.942466 13245 tablet_bootstrap.cc:492] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.944267 13245 tablet_bootstrap.cc:654] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.946620 13249 ts_tablet_manager.cc:1398] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.090s user 0.003s sys 0.000s
I20240506 19:56:34.947163 13249 tablet_bootstrap.cc:492] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:34.948971 13249 tablet_bootstrap.cc:654] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.950268 12825 tablet_service.cc:1467] Processing CreateTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 (DEFAULT_TABLE table=default.Table_038664bd22da4e20b40b21843516298f [id=34aeae2e9b5546d1b405ce1432e3c0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.950887 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a8b12f36e74c95ad6eaf30e432f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.953075 13245 tablet_bootstrap.cc:492] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.953347 13245 ts_tablet_manager.cc:1367] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240506 19:56:34.954437 13245 raft_consensus.cc:348] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.954782 13245 raft_consensus.cc:374] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.954879 13245 raft_consensus.cc:729] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.955250 13245 consensus_queue.cc:260] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.956189 13245 ts_tablet_manager.cc:1398] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:34.956578 13245 tablet_bootstrap.cc:492] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.958317 13101 tablet_service.cc:1467] Processing CreateTablet for tablet d05c5ca7ff864fa98f3494032729e978 (DEFAULT_TABLE table=default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.958730 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05c5ca7ff864fa98f3494032729e978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.958890 13245 tablet_bootstrap.cc:654] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.966013 13245 tablet_bootstrap.cc:492] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:34.966233 13245 ts_tablet_manager.cc:1367] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240506 19:56:34.966611 13103 tablet_service.cc:1467] Processing CreateTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad (DEFAULT_TABLE table=default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc [id=49660c755f8f470b8b8dc301ca27bb26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.967036 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e7329b22414da886dee4e1ec6bf3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.967232 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f (DEFAULT_TABLE table=default.Table_3a86534bf1014d9890190f09491e899d [id=d913e8d66e1b47f7a69d9a48d03c1722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.967552 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c1590e36b4471d8e72f4b07410ee0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.967446 13245 raft_consensus.cc:348] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.967772 13245 raft_consensus.cc:374] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.967911 13245 raft_consensus.cc:729] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.968240 13245 consensus_queue.cc:260] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:34.969341 13245 ts_tablet_manager.cc:1398] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240506 19:56:34.969743 13245 tablet_bootstrap.cc:492] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:34.970914 13094 tablet_service.cc:1467] Processing CreateTablet for tablet b5a443c0d35741b9966b32a7808a97e2 (DEFAULT_TABLE table=default.Table_93498ac3d84b4de4af0e703aa2b03878 [id=54286945a7554349b48aa6511f84326c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.971299 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a443c0d35741b9966b32a7808a97e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.972062 13245 tablet_bootstrap.cc:654] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.966581 13101 tablet_service.cc:1467] Processing CreateTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 (DEFAULT_TABLE table=default.Table_038664bd22da4e20b40b21843516298f [id=34aeae2e9b5546d1b405ce1432e3c0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.973556 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3a86534bf1014d9890190f09491e899d" } new_table_name: "default.Table_1a0e2e4b842f4375ab63b84f691c796b" modify_external_catalogs: true
I20240506 19:56:34.973922 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a8b12f36e74c95ad6eaf30e432f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.975121 13091 tablet_service.cc:1467] Processing CreateTablet for tablet cc8335f9308046279a25a4a95d8e0f65 (DEFAULT_TABLE table=default.Table_df1733520c4a4ae6a99a78f194d6cf06 [id=64a093df5c7143668da7d3e91eccba2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.975463 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8335f9308046279a25a4a95d8e0f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.977523 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5bce626686754b56b9573ca9293b568f" } new_table_name: "default.Table_291e179ddb654a8e90d5e5ea4db36e60" modify_external_catalogs: true
I20240506 19:56:34.983680 13092 tablet_service.cc:1467] Processing CreateTablet for tablet d3ec577dd06640068575288c24e1105a (DEFAULT_TABLE table=default.Table_b0e5bbcbcb324f54bc4feab7c750c029 [id=436b5c4427c24f4caf61a5c06a923b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.984099 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ec577dd06640068575288c24e1105a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.986217 13249 tablet_bootstrap.cc:492] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:34.986397 13249 ts_tablet_manager.cc:1367] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.002s
I20240506 19:56:34.987192 13249 raft_consensus.cc:348] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.987411 13249 raft_consensus.cc:374] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:34.987486 13249 raft_consensus.cc:729] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:34.987758 13249 consensus_queue.cc:260] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:34.988505 13249 ts_tablet_manager.cc:1398] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240506 19:56:34.988827 13249 tablet_bootstrap.cc:492] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:34.990470 13249 tablet_bootstrap.cc:654] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:34.990963 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 903ce1bc6ae7488d83bb0945424ed9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:34.991365 13242 tablet_replica.cc:330] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:34.991497 12826 tablet_service.cc:1467] Processing CreateTablet for tablet aae313961a914ed485f866d85785a788 (DEFAULT_TABLE table=default.Table_86cbff0af1184af6a66af4ee25521f9d [id=a63e1c9e058a4501831a67516affe578]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:34.991675 13242 raft_consensus.cc:2229] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:34.991897 13242 raft_consensus.cc:2258] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:34.991952 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae313961a914ed485f866d85785a788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:34.992661 13242 ts_tablet_manager.cc:1875] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:35.000440 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acd3fc62e9614cf3bde0202ee6832bb3 consensus state has no leader
I20240506 19:56:35.003360 13242 ts_tablet_manager.cc:1888] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.003568 13242 log.cc:1196] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/903ce1bc6ae7488d83bb0945424ed9f9
I20240506 19:56:35.004026 13242 ts_tablet_manager.cc:1909] T 903ce1bc6ae7488d83bb0945424ed9f9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.005468 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 903ce1bc6ae7488d83bb0945424ed9f9 (table default.Table_675510c53e1848ed8176a861a1ace8c4 [id=2b7d2dfedf4d4169a2244e104dcd154d]) successfully deleted
I20240506 19:56:35.006057 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5c4383f23b354870b6fddac21d4a453a" } new_table_name: "default.Table_eba34a76ef98415ea1596d0f5bd50b7f" modify_external_catalogs: true
I20240506 19:56:35.007637 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 09856d2dd66043fbab08966d04e7b992 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.008913 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 09856d2dd66043fbab08966d04e7b992 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.009917 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_209a7bac6fc84e80afea650dd487e1e0" } new_table_name: "default.Table_cef67076343c4a73b29b2d8091257dd2" modify_external_catalogs: true
I20240506 19:56:35.010634 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ab4e6c606a634df58f2180361078d3d5 (table default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]) successfully deleted
I20240506 19:56:35.010490 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9321b5f2115c4ff8aa68a30db878b09c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.010984 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f66f1e46cce47639fe644fb4ed2e5c0 already in progress: creating tablet
W20240506 19:56:35.011087 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9321b5f2115c4ff8aa68a30db878b09c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.011219 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 6d8a9ecf47d7474c82758417449d940c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d8a9ecf47d7474c82758417449d940c already in progress: creating tablet
W20240506 19:56:35.011457 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 30d819d1237f4798b4847fdf378cbc62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30d819d1237f4798b4847fdf378cbc62 already in progress: creating tablet
I20240506 19:56:35.011655 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e5789654f78a46f79a248c9ed47f314f" } new_table_name: "default.Table_c3908be635fe46a6a6efe79f7916080d" modify_external_catalogs: true
I20240506 19:56:35.013224 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9ec35e469b0e4d8ca1557a818fd7bee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.013964 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ec35e469b0e4d8ca1557a818fd7bee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.020202 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20c1590e36b4471d8e72f4b07410ee0f consensus state has no leader
I20240506 19:56:35.020566 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc" } modify_external_catalogs: true
I20240506 19:56:35.020766 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc" } modify_external_catalogs: true
W20240506 19:56:35.011677 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 27b24fa4663244a88dac9ea99fe38f3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b24fa4663244a88dac9ea99fe38f3a already in progress: creating tablet
I20240506 19:56:35.022606 13329 raft_consensus.cc:482] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.022753 13329 raft_consensus.cc:504] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.023478 13329 leader_election.cc:288] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:35.023674 13244 tablet_bootstrap.cc:492] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.023912 13244 ts_tablet_manager.cc:1367] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.274s user 0.004s sys 0.000s
I20240506 19:56:35.024183 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25ba6b9b24a48afb1c6f2ba8d6fd42b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:35.024390 12988 raft_consensus.cc:2454] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:35.025101 13057 leader_election.cc:302] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:35.025038 13244 raft_consensus.cc:348] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:35.025358 13244 raft_consensus.cc:374] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.025460 13244 raft_consensus.cc:729] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.025839 13244 consensus_queue.cc:260] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:35.026096 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25ba6b9b24a48afb1c6f2ba8d6fd42b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:35.026753 13058 leader_election.cc:341] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.026852 13249 tablet_bootstrap.cc:492] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.026986 13249 ts_tablet_manager.cc:1367] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240506 19:56:35.027235 13329 raft_consensus.cc:2781] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:35.027348 13329 raft_consensus.cc:482] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:35.027441 13329 raft_consensus.cc:3037] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:35.027966 13249 raft_consensus.cc:348] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.028167 13249 raft_consensus.cc:374] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.028242 13249 raft_consensus.cc:729] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.028463 13249 consensus_queue.cc:260] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.029011 13249 ts_tablet_manager.cc:1398] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:35.029626 13249 tablet_bootstrap.cc:492] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.030073 13244 ts_tablet_manager.cc:1398] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240506 19:56:35.031770 13249 tablet_bootstrap.cc:654] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.031821 13244 tablet_bootstrap.cc:492] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.034039 13244 tablet_bootstrap.cc:654] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.040395 13249 tablet_bootstrap.cc:492] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.040604 13249 ts_tablet_manager.cc:1367] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240506 19:56:35.041473 13249 raft_consensus.cc:348] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.041721 13249 raft_consensus.cc:374] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.041806 13249 raft_consensus.cc:729] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.042078 13244 tablet_bootstrap.cc:492] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.042083 13249 consensus_queue.cc:260] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.042284 13244 ts_tablet_manager.cc:1367] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240506 19:56:35.042856 13249 ts_tablet_manager.cc:1398] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:35.043195 13249 tablet_bootstrap.cc:492] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.044545 13244 raft_consensus.cc:348] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:35.044842 13249 tablet_bootstrap.cc:654] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.044837 13244 raft_consensus.cc:374] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.044941 13244 raft_consensus.cc:729] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:35.045225 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1667a0bbd3af4d678434b068c47cc518 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1667a0bbd3af4d678434b068c47cc518 already in progress: creating tablet
W20240506 19:56:35.045600 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1488328703024ae8983add9e4d6afc8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1488328703024ae8983add9e4d6afc8f already in progress: creating tablet
I20240506 19:56:35.046540 13244 consensus_queue.cc:260] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:56:35.046752 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 0ca44a4cb8344307b94c8fafba143836 (table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]) successfully deleted
W20240506 19:56:35.047008 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d843fe82af1148f4a9045ce8b09ad11d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d843fe82af1148f4a9045ce8b09ad11d already in progress: creating tablet
I20240506 19:56:35.047070 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bec645fde8cf480cb56580e5edcd5d76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.047554 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a16704567ef6454eaef886fb9cd599ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a16704567ef6454eaef886fb9cd599ce already in progress: creating tablet
I20240506 19:56:35.047746 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d2abecd9ec7e4606b38c7d22477ddf65 (table default.Table_eb361931fc51481590136c98bc041bc6 [id=031d0b422342478ba95396f085780f3b]) successfully deleted
I20240506 19:56:35.048784 13329 raft_consensus.cc:504] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.049156 13244 ts_tablet_manager.cc:1398] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240506 19:56:35.050262 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25ba6b9b24a48afb1c6f2ba8d6fd42b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:35.051157 13058 leader_election.cc:341] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.051398 13244 tablet_bootstrap.cc:492] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:35.052258 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 89eb4e2701894f32a85d2cd89da58ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89eb4e2701894f32a85d2cd89da58ec2 already in progress: creating tablet
I20240506 19:56:35.052492 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 903ce1bc6ae7488d83bb0945424ed9f9 (table default.Table_675510c53e1848ed8176a861a1ace8c4 [id=2b7d2dfedf4d4169a2244e104dcd154d]) successfully deleted
I20240506 19:56:35.052448 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_38260663814f40939e221b4885ed23a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.052706 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b529bb9362924caca399491e7d9e03cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b529bb9362924caca399491e7d9e03cd already in progress: creating tablet
W20240506 19:56:35.052913 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2502b423bd1040539fb1a3ebbece3c1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2502b423bd1040539fb1a3ebbece3c1c already in progress: creating tablet
W20240506 19:56:35.053009 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38260663814f40939e221b4885ed23a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.053596 13244 tablet_bootstrap.cc:654] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:35.054180 12508 consensus_peers.cc:485] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d -> Peer 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819): Couldn't send request to peer 27d4482cd59a4e5687992b1f8a177373. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240506 19:56:35.055717 12507 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet acd3fc62e9614cf3bde0202ee6832bb3 consensus state has no leader
W20240506 19:56:35.057763 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bec645fde8cf480cb56580e5edcd5d76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.059463 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 (DEFAULT_TABLE table=default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.059967 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00de09ddcdd948a6a3500a6c9d814c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.060034 13244 tablet_bootstrap.cc:492] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.060252 13244 ts_tablet_manager.cc:1367] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240506 19:56:35.061364 13244 raft_consensus.cc:348] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.061717 13244 raft_consensus.cc:374] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.061818 13244 raft_consensus.cc:729] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.065331 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25ba6b9b24a48afb1c6f2ba8d6fd42b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:35.065575 12988 raft_consensus.cc:3037] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:35.065706 12491 master_runner.cc:186] Error getting master registration for 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111),
I20240506 19:56:35.065871 12491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 19:56:35.066431 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.066716 13246 tablet_replica.cc:330] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.067205 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 10db9b8cad5045309dff3751cefce838 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.067783 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet b63857fa6004468cad193ed4f2a9d366 with delete_type TABLET_DATA_DELETED (replaced by tablet 64b7e42061cd487b8517447eb59ea8a1) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.068001 13246 raft_consensus.cc:2229] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.068181 13246 raft_consensus.cc:2258] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.068580 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet f017e635010040008be72b3813b5a4ed with delete_type TABLET_DATA_DELETED (replaced by tablet 4a080ccac6394210b0b24b65843f9d75) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.068930 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b with delete_type TABLET_DATA_DELETED (replaced by tablet 959f9e652bdd430898c1d3a95fe765f6) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.069285 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 18920e08b0de4f5292abb54e3432ee39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.069509 13244 consensus_queue.cc:260] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.069737 12988 raft_consensus.cc:2454] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:35.070672 13246 ts_tablet_manager.cc:1875] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.073956 13249 tablet_bootstrap.cc:492] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.073953 13057 leader_election.cc:302] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.074106 13249 ts_tablet_manager.cc:1367] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.006s
I20240506 19:56:35.075054 13249 raft_consensus.cc:348] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.075263 13249 raft_consensus.cc:374] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.075349 13249 raft_consensus.cc:729] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.075651 13249 consensus_queue.cc:260] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.076021 13244 ts_tablet_manager.cc:1398] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
I20240506 19:56:35.076275 13249 ts_tablet_manager.cc:1398] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:35.076583 13249 tablet_bootstrap.cc:492] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.078035 13244 tablet_bootstrap.cc:492] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.078478 13249 tablet_bootstrap.cc:654] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.080174 13244 tablet_bootstrap.cc:654] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.096705 13329 leader_election.cc:288] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:35.125537 13329 raft_consensus.cc:2781] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:35.125926 13329 raft_consensus.cc:686] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:35.126452 13329 consensus_queue.cc:237] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.145220 13246 ts_tablet_manager.cc:1888] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.145597 13246 log.cc:1196] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b5adab0af6274ba0ad54402c5b1ff757
I20240506 19:56:35.146239 13246 ts_tablet_manager.cc:1909] T b5adab0af6274ba0ad54402c5b1ff757 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.147905 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b5adab0af6274ba0ad54402c5b1ff757 (table default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 [id=48be0d45bbf34e6187819fd3994a0afe]) successfully deleted
W20240506 19:56:35.155167 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 81236.7us
I20240506 19:56:35.162081 13313 server_base.cc:1034] running on GCE node
I20240506 19:56:35.162775 13246 tablet_replica.cc:330] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.163187 13246 raft_consensus.cc:2229] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.163552 13246 raft_consensus.cc:2258] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:35.164355 13383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:35.165701 13313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:35.166024 13246 ts_tablet_manager.cc:1875] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.170709 13313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:56:35.174064 13313 hybrid_clock.cc:648] HybridClock initialized: now 1715025395173944 us; error 91 us; skew 500 ppm
I20240506 19:56:35.174515 13313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:56:35.175446 13246 ts_tablet_manager.cc:1888] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.182250 13249 tablet_bootstrap.cc:492] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.183053 13249 ts_tablet_manager.cc:1367] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.106s user 0.037s sys 0.062s
I20240506 19:56:35.183918 13249 raft_consensus.cc:348] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.184285 13249 raft_consensus.cc:374] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.184398 13249 raft_consensus.cc:729] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.184791 13249 consensus_queue.cc:260] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.186981 13244 tablet_bootstrap.cc:492] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.187155 13244 ts_tablet_manager.cc:1367] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.109s user 0.022s sys 0.027s
I20240506 19:56:35.187471 13249 ts_tablet_manager.cc:1398] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:35.188014 13244 raft_consensus.cc:348] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.188246 13244 raft_consensus.cc:374] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.188349 13244 raft_consensus.cc:729] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.188880 13249 tablet_bootstrap.cc:492] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:35.189465 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2751074a54ed4bcea6979e6ad11c2efe consensus state has no leader
W20240506 19:56:35.189596 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dc0ac37f67942fc8a31d8d43c839246 consensus state has no leader
I20240506 19:56:35.190443 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 09856d2dd66043fbab08966d04e7b992
I20240506 19:56:35.191088 12522 catalog_manager.cc:6554] ReplaceTablet: tablet 09856d2dd66043fbab08966d04e7b992 deleted and replaced by tablet 6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:35.191424 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 09856d2dd66043fbab08966d04e7b992 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d04a1c5ee7b4d529fa2f4da4945ff09) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.191951 13249 tablet_bootstrap.cc:654] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.192940 13244 consensus_queue.cc:260] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:35.193513 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 09856d2dd66043fbab08966d04e7b992 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09856d2dd66043fbab08966d04e7b992 already in progress: creating tablet
I20240506 19:56:35.194572 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 09856d2dd66043fbab08966d04e7b992 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d04a1c5ee7b4d529fa2f4da4945ff09) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:35.195406 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 09856d2dd66043fbab08966d04e7b992 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09856d2dd66043fbab08966d04e7b992 already in progress: creating tablet
I20240506 19:56:35.199986 13244 ts_tablet_manager.cc:1398] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240506 19:56:35.200381 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 30cb3cb7cbe14cc9bf1bfd07227f538e
W20240506 19:56:35.200599 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1f1e4efbe8b40b199fcb3020569e1a1 consensus state has no leader
I20240506 19:56:35.201977 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:35.205358 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30cb3cb7cbe14cc9bf1bfd07227f538e already in progress: creating tablet
I20240506 19:56:35.204942 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:35.206938 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30cb3cb7cbe14cc9bf1bfd07227f538e already in progress: creating tablet
I20240506 19:56:35.209239 13244 tablet_bootstrap.cc:492] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.213213 13244 tablet_bootstrap.cc:654] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.220957 13249 tablet_bootstrap.cc:492] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.221151 13249 ts_tablet_manager.cc:1367] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.011s
I20240506 19:56:35.222033 13249 raft_consensus.cc:348] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.222254 13249 raft_consensus.cc:374] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.222366 13249 raft_consensus.cc:729] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.222707 13313 webserver.cc:460] Webserver started at http://127.12.48.253:35887/ using document root <none> and password file <none>
I20240506 19:56:35.222638 13249 consensus_queue.cc:260] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.224171 13313 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:35.224314 13249 ts_tablet_manager.cc:1398] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240506 19:56:35.224462 13313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:35.225451 13249 tablet_bootstrap.cc:492] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.227118 13249 tablet_bootstrap.cc:654] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.234848 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet c5e3f798d6484d4fabd55f9b4c98a911 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.242192 13244 tablet_bootstrap.cc:492] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.242388 13244 ts_tablet_manager.cc:1367] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.002s
I20240506 19:56:35.244158 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6da5f3268c67468f810ffaaa56d3cfdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.244812 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8b16088d777a4dcfb6b6d4be569babec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.245736 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b53ac51ea36943aa94434bfe07d4dfa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.246268 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet a16704567ef6454eaef886fb9cd599ce from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:35.246491 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b53ac51ea36943aa94434bfe07d4dfa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.246551 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62612e9478664e9d86a80129badd6c12" } modify_external_catalogs: true
I20240506 19:56:35.246737 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62612e9478664e9d86a80129badd6c12" } modify_external_catalogs: true
I20240506 19:56:35.246789 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4b6f407e4a464024aed5a1fb84c49849" } modify_external_catalogs: true
I20240506 19:56:35.246961 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4b6f407e4a464024aed5a1fb84c49849" } modify_external_catalogs: true
I20240506 19:56:35.248234 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c905679471c7485fa59180f110b5ec1f" } modify_external_catalogs: true
I20240506 19:56:35.248461 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c905679471c7485fa59180f110b5ec1f" } modify_external_catalogs: true
I20240506 19:56:35.249125 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet c51b52a98cec48fc959b01a2a0a8795c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.249504 13242 tablet_replica.cc:330] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.249697 13242 raft_consensus.cc:2229] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.249837 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5513f4926e41a6b78f02a5126cc13a with delete_type TABLET_DATA_DELETED (replaced by tablet 2751074a54ed4bcea6979e6ad11c2efe) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.249887 13242 raft_consensus.cc:2258] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.250526 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cd0e12d470dd429295bcb3d354afacfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.250663 13242 ts_tablet_manager.cc:1875] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.251041 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ca557811b003457fbf1621e8ca55a53d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.251439 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b16088d777a4dcfb6b6d4be569babec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.251549 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca557811b003457fbf1621e8ca55a53d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.252727 13353 raft_consensus.cc:482] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.252883 13353 raft_consensus.cc:504] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.253620 13353 leader_election.cc:288] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:35.254472 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44de7153689b4dbd80dae87d3ee48a83" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:35.255637 12921 leader_election.cc:341] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:35.251039 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd0e12d470dd429295bcb3d354afacfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.257316 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da5f3268c67468f810ffaaa56d3cfdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.258085 13242 ts_tablet_manager.cc:1888] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.258757 13242 log.cc:1196] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/c51b52a98cec48fc959b01a2a0a8795c
I20240506 19:56:35.258761 12514 catalog_manager.cc:5484] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:35.259166 13242 ts_tablet_manager.cc:1909] T c51b52a98cec48fc959b01a2a0a8795c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.259346 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:35.260004 13244 raft_consensus.cc:348] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.260221 13244 raft_consensus.cc:374] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.260299 13244 raft_consensus.cc:729] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.260560 13244 consensus_queue.cc:260] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.261826 13249 tablet_bootstrap.cc:492] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.262007 13249 ts_tablet_manager.cc:1367] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.008s
I20240506 19:56:35.262722 13244 ts_tablet_manager.cc:1398] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240506 19:56:35.263918 13246 log.cc:1196] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/10db9b8cad5045309dff3751cefce838
I20240506 19:56:35.264915 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet c51b52a98cec48fc959b01a2a0a8795c (table default.Table_2a4c70f40666495ba552107168859093 [id=513f5de1c20e4aa48bd9f216c869b5fe]) successfully deleted
I20240506 19:56:35.266899 13244 tablet_bootstrap.cc:492] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.262768 13249 raft_consensus.cc:348] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.267293 13249 raft_consensus.cc:374] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.267383 13249 raft_consensus.cc:729] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.268182 13249 consensus_queue.cc:260] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.268656 13244 tablet_bootstrap.cc:654] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.268738 13249 ts_tablet_manager.cc:1398] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:35.270742 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b38453bf678748399339806d12993c88
I20240506 19:56:35.270879 13249 tablet_bootstrap.cc:492] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.271540 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet b38453bf678748399339806d12993c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.272140 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet b38453bf678748399339806d12993c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:35.272897 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b38453bf678748399339806d12993c88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b38453bf678748399339806d12993c88 already in progress: creating tablet
I20240506 19:56:35.273105 13242 tablet_replica.cc:330] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.273798 13242 raft_consensus.cc:2229] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.274051 13242 raft_consensus.cc:2258] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.274883 13242 ts_tablet_manager.cc:1875] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.264950 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 09856d2dd66043fbab08966d04e7b992 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d04a1c5ee7b4d529fa2f4da4945ff09) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.263950 13400 raft_consensus.cc:482] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.277861 13400 raft_consensus.cc:504] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.278873 13400 leader_election.cc:288] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:35.265419 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.265517 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44de7153689b4dbd80dae87d3ee48a83" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:35.272365 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet b38453bf678748399339806d12993c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.277339 13249 tablet_bootstrap.cc:654] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.280493 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f017e635010040008be72b3813b5a4ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:35.280956 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f017e635010040008be72b3813b5a4ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:35.281224 12988 raft_consensus.cc:2454] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:35.282327 13057 leader_election.cc:302] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:35.283531 12922 leader_election.cc:341] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.283706 12922 leader_election.cc:302] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.284217 13353 raft_consensus.cc:2726] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:35.284616 13058 leader_election.cc:341] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.285064 13400 raft_consensus.cc:2781] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:35.285194 13400 raft_consensus.cc:482] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:35.285282 13400 raft_consensus.cc:3037] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:35.288229 13353 raft_consensus.cc:482] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.288445 13353 raft_consensus.cc:504] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.289192 13353 leader_election.cc:288] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.290302 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5adab0af6274ba0ad54402c5b1ff757" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:35.290900 12921 leader_election.cc:341] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.291467 13313 fs_manager.cc:709] Time spent opening directory manager: real 0.059s user 0.006s sys 0.002s
I20240506 19:56:35.292236 13400 raft_consensus.cc:504] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:35.292683 12922 leader_election.cc:341] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b5adab0af6274ba0ad54402c5b1ff757
I20240506 19:56:35.292825 12922 leader_election.cc:302] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.293331 13353 raft_consensus.cc:2726] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:35.295142 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f017e635010040008be72b3813b5a4ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:35.296078 13058 leader_election.cc:341] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.297138 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f017e635010040008be72b3813b5a4ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:35.297422 12988 raft_consensus.cc:3037] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:35.298833 13242 ts_tablet_manager.cc:1888] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.299006 13242 log.cc:1196] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1d5513f4926e41a6b78f02a5126cc13a
I20240506 19:56:35.299396 13242 ts_tablet_manager.cc:1909] T 1d5513f4926e41a6b78f02a5126cc13a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.300562 12988 raft_consensus.cc:2454] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:35.300565 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1d5513f4926e41a6b78f02a5126cc13a (table default.Table_cef67076343c4a73b29b2d8091257dd2 [id=1ef16dfbd249466fa6b0a7df43515b7b]) successfully deleted
I20240506 19:56:35.301656 13057 leader_election.cc:302] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.302064 13329 raft_consensus.cc:2781] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
W20240506 19:56:35.304644 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b38453bf678748399339806d12993c88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b38453bf678748399339806d12993c88 already in progress: creating tablet
I20240506 19:56:35.304733 13400 leader_election.cc:288] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:35.277525 13245 tablet_bootstrap.cc:492] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.307883 13245 ts_tablet_manager.cc:1367] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.338s user 0.070s sys 0.125s
I20240506 19:56:35.310719 13245 raft_consensus.cc:348] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.311471 13245 raft_consensus.cc:374] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.311911 13245 raft_consensus.cc:729] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.313309 13246 ts_tablet_manager.cc:1909] T 10db9b8cad5045309dff3751cefce838 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.312520 13245 consensus_queue.cc:260] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.319566 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fdae8810fa204843b7e162d421c80856" } modify_external_catalogs: true
I20240506 19:56:35.319828 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fdae8810fa204843b7e162d421c80856" } modify_external_catalogs: true
I20240506 19:56:35.319912 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet b63857fa6004468cad193ed4f2a9d366 with delete_type TABLET_DATA_DELETED (replaced by tablet 64b7e42061cd487b8517447eb59ea8a1) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.320223 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca44a4cb8344307b94c8fafba143836 with delete_type TABLET_DATA_DELETED (replaced by tablet 92951843c8bd446aa86436b65d9eb793) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.320495 13242 tablet_replica.cc:330] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.320684 13242 raft_consensus.cc:2229] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.320878 13242 raft_consensus.cc:2258] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.321686 13242 ts_tablet_manager.cc:1875] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.322463 13353 raft_consensus.cc:482] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.322656 13353 raft_consensus.cc:504] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.323357 13353 leader_election.cc:288] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.325714 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e896b3ba7c4864a26cdfce76fadcfb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:35.326326 12921 leader_election.cc:341] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.327064 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e896b3ba7c4864a26cdfce76fadcfb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:35.327536 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c5e3f798d6484d4fabd55f9b4c98a911
W20240506 19:56:35.327848 12922 leader_election.cc:341] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.327980 12922 leader_election.cc:302] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.328150 12513 catalog_manager.cc:6554] ReplaceTablet: tablet c5e3f798d6484d4fabd55f9b4c98a911 deleted and replaced by tablet a2ff5fed57d64244894f4af1f6067e85
I20240506 19:56:35.328788 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff5fed57d64244894f4af1f6067e85) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.328933 13242 ts_tablet_manager.cc:1888] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.328969 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d6c656e16c1045dea79bc23b64fdd5e8
I20240506 19:56:35.329042 13242 log.cc:1196] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b63857fa6004468cad193ed4f2a9d366
W20240506 19:56:35.329751 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c5e3f798d6484d4fabd55f9b4c98a911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5e3f798d6484d4fabd55f9b4c98a911 already in progress: creating tablet
I20240506 19:56:35.330332 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.330718 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff5fed57d64244894f4af1f6067e85) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.330987 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.331972 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff5fed57d64244894f4af1f6067e85) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:35.332026 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d6c656e16c1045dea79bc23b64fdd5e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c656e16c1045dea79bc23b64fdd5e8 already in progress: creating tablet
I20240506 19:56:35.332237 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.332607 13353 raft_consensus.cc:2726] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:35.334311 13245 ts_tablet_manager.cc:1398] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:35.334486 13242 ts_tablet_manager.cc:1909] T b63857fa6004468cad193ed4f2a9d366 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.335368 13245 tablet_bootstrap.cc:492] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.335645 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b63857fa6004468cad193ed4f2a9d366 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]) successfully deleted
I20240506 19:56:35.336145 13242 tablet_replica.cc:330] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.336449 13242 raft_consensus.cc:2229] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.336722 13242 raft_consensus.cc:2258] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.336869 13244 tablet_bootstrap.cc:492] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.337071 13244 ts_tablet_manager.cc:1367] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.009s
I20240506 19:56:35.337710 13242 ts_tablet_manager.cc:1875] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.338030 13244 raft_consensus.cc:348] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.338272 13244 raft_consensus.cc:374] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.338364 13244 raft_consensus.cc:729] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.338642 13244 consensus_queue.cc:260] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.339406 13244 ts_tablet_manager.cc:1398] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:35.339740 13244 tablet_bootstrap.cc:492] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.340685 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 09856d2dd66043fbab08966d04e7b992 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.341241 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_79cba7e264ba4a9e8b17244167924764" } new_table_name: "default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7" modify_external_catalogs: true
I20240506 19:56:35.342600 13246 tablet_replica.cc:330] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.342815 13246 raft_consensus.cc:2229] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.342989 13246 raft_consensus.cc:2258] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.343014 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a577b6ec9dc64f0f9be2a66998f740b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.343662 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.344586 13245 tablet_bootstrap.cc:654] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.345220 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 10db9b8cad5045309dff3751cefce838 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.346174 13244 tablet_bootstrap.cc:654] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.347110 13329 raft_consensus.cc:686] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:35.347744 13246 ts_tablet_manager.cc:1875] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.347680 13329 consensus_queue.cc:237] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.348799 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 10db9b8cad5045309dff3751cefce838 (table default.Table_06ec673eaab44720a193e5236fe7481d [id=5f25a2a3fe494e7f88d31b78f2e482c2]) successfully deleted
I20240506 19:56:35.357182 13242 ts_tablet_manager.cc:1888] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.357997 13418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:35.360389 13249 tablet_bootstrap.cc:492] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.360571 13249 ts_tablet_manager.cc:1367] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.027s
I20240506 19:56:35.361057 13245 tablet_bootstrap.cc:492] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.361356 13245 ts_tablet_manager.cc:1367] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240506 19:56:35.361443 13249 raft_consensus.cc:348] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.361680 13249 raft_consensus.cc:374] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.361783 13249 raft_consensus.cc:729] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.362051 13249 consensus_queue.cc:260] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.362612 13249 ts_tablet_manager.cc:1398] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:35.363401 13249 tablet_bootstrap.cc:492] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.364125 13246 ts_tablet_manager.cc:1888] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.364292 13246 log.cc:1196] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b63857fa6004468cad193ed4f2a9d366
I20240506 19:56:35.364627 13246 ts_tablet_manager.cc:1909] T b63857fa6004468cad193ed4f2a9d366 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.365599 13249 tablet_bootstrap.cc:654] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.367054 13313 fs_manager.cc:725] Time spent opening block manager: real 0.070s user 0.010s sys 0.003s
I20240506 19:56:35.367053 13353 consensus_queue.cc:1035] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240506 19:56:35.367090 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c715baa1d25447d991935176e0f72c6b" } new_table_name: "default.Table_bb529dd8d37f40449f27b0e07cbc93dc" modify_external_catalogs: true
I20240506 19:56:35.367254 13313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
uuid: "27d4482cd59a4e5687992b1f8a177373"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
I20240506 19:56:35.370249 13313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:35.373915 13245 raft_consensus.cc:348] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.374153 13245 raft_consensus.cc:374] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.374256 13245 raft_consensus.cc:729] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.376896 13245 consensus_queue.cc:260] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.378000 13249 tablet_bootstrap.cc:492] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.378196 13249 ts_tablet_manager.cc:1367] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20240506 19:56:35.358743 13244 tablet_bootstrap.cc:492] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.377250 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4e6c606a634df58f2180361078d3d5 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e3260f46024386baac64cb8677883c) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.379068 13249 raft_consensus.cc:348] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.379382 13249 raft_consensus.cc:374] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.379392 13244 ts_tablet_manager.cc:1367] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240506 19:56:35.379477 13249 raft_consensus.cc:729] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.380103 13249 consensus_queue.cc:260] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.380432 13244 raft_consensus.cc:348] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.380712 13244 raft_consensus.cc:374] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.380761 13249 ts_tablet_manager.cc:1398] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:35.380831 13244 raft_consensus.cc:729] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.381217 13249 tablet_bootstrap.cc:492] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.381207 13244 consensus_queue.cc:260] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.382681 13245 ts_tablet_manager.cc:1398] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240506 19:56:35.383098 13244 ts_tablet_manager.cc:1398] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:35.384392 13244 tablet_bootstrap.cc:492] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.386161 13244 tablet_bootstrap.cc:654] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.386415 13245 tablet_bootstrap.cc:492] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.387748 13249 tablet_bootstrap.cc:654] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.386358 13242 log.cc:1196] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/0ca44a4cb8344307b94c8fafba143836
I20240506 19:56:35.388723 13245 tablet_bootstrap.cc:654] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.389218 13242 ts_tablet_manager.cc:1909] T 0ca44a4cb8344307b94c8fafba143836 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.390571 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b63857fa6004468cad193ed4f2a9d366 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]) successfully deleted
I20240506 19:56:35.391094 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 0ca44a4cb8344307b94c8fafba143836 (table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]) successfully deleted
I20240506 19:56:35.392510 13246 tablet_replica.cc:330] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.392706 13246 raft_consensus.cc:2229] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:35.393034 13246 pending_rounds.cc:62] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:35.393118 13246 pending_rounds.cc:68] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744019347656704 op_type: NO_OP noop_request { }
I20240506 19:56:35.394372 13246 raft_consensus.cc:2866] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:35.394502 13246 raft_consensus.cc:2258] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.395752 13249 tablet_bootstrap.cc:492] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.395924 13249 ts_tablet_manager.cc:1367] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
W20240506 19:56:35.396138 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c5e3f798d6484d4fabd55f9b4c98a911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5e3f798d6484d4fabd55f9b4c98a911 already in progress: creating tablet
I20240506 19:56:35.396821 13249 raft_consensus.cc:348] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.397042 13249 raft_consensus.cc:374] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.397150 13249 raft_consensus.cc:729] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.397395 13249 consensus_queue.cc:260] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:35.397859 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d6c656e16c1045dea79bc23b64fdd5e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c656e16c1045dea79bc23b64fdd5e8 already in progress: creating tablet
I20240506 19:56:35.397950 13249 ts_tablet_manager.cc:1398] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:35.397933 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_32ef24b21b0a4018b9baabda7d2e552b" } new_table_name: "default.Table_c7ba3500fa6c464996810bc6a6a1af48" modify_external_catalogs: true
I20240506 19:56:35.398243 13249 tablet_bootstrap.cc:492] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.400727 13249 tablet_bootstrap.cc:654] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.402243 13242 tablet_replica.cc:330] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.402511 13242 raft_consensus.cc:2229] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.402755 13242 raft_consensus.cc:2258] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.403609 13242 ts_tablet_manager.cc:1875] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.408911 13249 tablet_bootstrap.cc:492] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.409107 13249 ts_tablet_manager.cc:1367] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240506 19:56:35.410007 13249 raft_consensus.cc:348] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.410261 13249 raft_consensus.cc:374] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.410346 13249 raft_consensus.cc:729] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.410616 13249 consensus_queue.cc:260] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.414644 13246 ts_tablet_manager.cc:1875] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.419031 13242 ts_tablet_manager.cc:1888] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.419337 13242 log.cc:1196] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/10db9b8cad5045309dff3751cefce838
I20240506 19:56:35.420017 13242 ts_tablet_manager.cc:1909] T 10db9b8cad5045309dff3751cefce838 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.420311 13244 tablet_bootstrap.cc:492] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.420784 13244 ts_tablet_manager.cc:1367] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20240506 19:56:35.421828 13245 tablet_bootstrap.cc:492] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.422024 13245 ts_tablet_manager.cc:1367] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.001s
I20240506 19:56:35.422324 13244 raft_consensus.cc:348] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.422632 13244 raft_consensus.cc:374] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.422775 13244 raft_consensus.cc:729] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.422835 13245 raft_consensus.cc:348] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.423069 13245 raft_consensus.cc:374] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.423158 13245 raft_consensus.cc:729] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.423302 13244 consensus_queue.cc:260] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.425410 12630 master_runner.cc:186] Error getting master registration for 127.12.48.253:36819: Network error: Client connection negotiation failed: client connection to 127.12.48.253:36819: connect: Connection refused (error 111),
I20240506 19:56:35.425633 12630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 19:56:35.427553 13244 ts_tablet_manager.cc:1398] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:35.428933 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_47a20691cc45428695761f4e7ef2e4fc" } new_table_name: "default.Table_fd07c6b21a494057a0f71bb60e892972" modify_external_catalogs: true
I20240506 19:56:35.430238 13244 tablet_bootstrap.cc:492] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.430238 13245 consensus_queue.cc:260] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.432039 13244 tablet_bootstrap.cc:654] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.432168 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:35.433055 13245 ts_tablet_manager.cc:1398] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240506 19:56:35.432960 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.433565 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.433969 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 10db9b8cad5045309dff3751cefce838 (table default.Table_06ec673eaab44720a193e5236fe7481d [id=5f25a2a3fe494e7f88d31b78f2e482c2]) successfully deleted
I20240506 19:56:35.435356 13245 tablet_bootstrap.cc:492] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.435969 13242 tablet_replica.cc:330] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.436780 13242 raft_consensus.cc:2229] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.437361 13242 raft_consensus.cc:2258] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.439283 13249 ts_tablet_manager.cc:1398] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.030s user 0.005s sys 0.005s
I20240506 19:56:35.439330 13242 ts_tablet_manager.cc:1875] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.439692 13249 tablet_bootstrap.cc:492] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.440678 13244 tablet_bootstrap.cc:492] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.440827 13244 ts_tablet_manager.cc:1367] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240506 19:56:35.441591 13244 raft_consensus.cc:348] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.441819 13244 raft_consensus.cc:374] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.441895 13244 raft_consensus.cc:729] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.443847 13434 consensus_queue.cc:1035] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240506 19:56:35.446151 13249 tablet_bootstrap.cc:654] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.446560 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4f91888e88234cbdbd00f48dc1574653
I20240506 19:56:35.449083 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_558320fe838a40a0af9f6e04106029b0" } new_table_name: "default.Table_fec5b763775747c9b4d63e7f0aed22ec" modify_external_catalogs: true
I20240506 19:56:35.449281 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:35.450330 13244 consensus_queue.cc:260] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.454003 13244 ts_tablet_manager.cc:1398] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240506 19:56:35.454479 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91888e88234cbdbd00f48dc1574653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.455122 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d08a9f7decde4bb69b28102c4b21a5bd" } modify_external_catalogs: true
I20240506 19:56:35.456583 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.457103 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdcdbb9cf3e4e9db6174bce7785565e with delete_type TABLET_DATA_DELETED (replaced by tablet a16704567ef6454eaef886fb9cd599ce) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.457540 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d08a9f7decde4bb69b28102c4b21a5bd" } modify_external_catalogs: true
W20240506 19:56:35.457612 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee59a76d52fc48d7bcf3256c359d3c02 consensus state has no leader
I20240506 19:56:35.458192 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4b69070c2c254336bfaec6ea640736d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.458829 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet d2abecd9ec7e4606b38c7d22477ddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.459017 13244 tablet_bootstrap.cc:492] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:35.459026 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b69070c2c254336bfaec6ea640736d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.460662 13244 tablet_bootstrap.cc:654] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:35.460681 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a080ccac6394210b0b24b65843f9d75 consensus state has no leader
I20240506 19:56:35.461019 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b7f56fea34946e590dae0621b28f9c6 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.462970 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 5516e8aad4e8474692b5b6c40bc483c1 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.464830 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f4146ffd1ac744c4bdd3506d398dcdc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.465523 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.465857 13244 tablet_bootstrap.cc:492] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.466032 13244 ts_tablet_manager.cc:1367] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240506 19:56:35.467023 13244 raft_consensus.cc:348] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.467238 13244 raft_consensus.cc:374] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.467329 13244 raft_consensus.cc:729] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.467734 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91888e88234cbdbd00f48dc1574653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.468230 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.468943 13242 ts_tablet_manager.cc:1888] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:35.469046 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3d57249f30a44dab3384092ec20b515 consensus state has no leader
I20240506 19:56:35.469071 13242 log.cc:1196] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ab4e6c606a634df58f2180361078d3d5
I20240506 19:56:35.469511 13242 ts_tablet_manager.cc:1909] T ab4e6c606a634df58f2180361078d3d5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.467775 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_970886a5e1a34021a2a6acf328821076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.470144 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_970886a5e1a34021a2a6acf328821076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.471436 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ab4e6c606a634df58f2180361078d3d5 (table default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]) successfully deleted
I20240506 19:56:35.471633 13245 tablet_bootstrap.cc:654] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.471899 13244 consensus_queue.cc:260] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.475006 13246 ts_tablet_manager.cc:1888] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:35.475565 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 18920e08b0de4f5292abb54e3432ee39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.476883 13246 log.cc:1196] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f017e635010040008be72b3813b5a4ed
I20240506 19:56:35.477319 13246 ts_tablet_manager.cc:1909] T f017e635010040008be72b3813b5a4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.477787 13244 ts_tablet_manager.cc:1398] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20240506 19:56:35.478600 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4f91888e88234cbdbd00f48dc1574653 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f91888e88234cbdbd00f48dc1574653 already in progress: creating tablet
I20240506 19:56:35.478933 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f017e635010040008be72b3813b5a4ed (table default.Table_bb529dd8d37f40449f27b0e07cbc93dc [id=4b9a2f4ea4b1460f89e8981e952e0d2d]) successfully deleted
W20240506 19:56:35.479982 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 72a3bed0eb234e21b890242e9d23a5b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a3bed0eb234e21b890242e9d23a5b4 already in progress: creating tablet
I20240506 19:56:35.481168 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca557811b003457fbf1621e8ca55a53d" } modify_external_catalogs: true
I20240506 19:56:35.481417 13245 tablet_bootstrap.cc:492] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.481510 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca557811b003457fbf1621e8ca55a53d" } modify_external_catalogs: true
I20240506 19:56:35.483314 13244 tablet_bootstrap.cc:492] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.484304 13242 tablet_replica.cc:330] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.486435 13245 ts_tablet_manager.cc:1367] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.004s
I20240506 19:56:35.487130 13434 raft_consensus.cc:482] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.487277 13245 raft_consensus.cc:348] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.487442 13434 raft_consensus.cc:504] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.488333 13434 leader_election.cc:288] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:35.488925 13246 tablet_replica.cc:330] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.489012 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79486583025c42bfb1321f9cb6dd96f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:35.489131 13246 raft_consensus.cc:2229] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:35.489234 13123 raft_consensus.cc:2454] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:35.489475 13246 pending_rounds.cc:62] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:35.489537 13246 pending_rounds.cc:68] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744018441986048 op_type: NO_OP noop_request { }
I20240506 19:56:35.489663 13246 raft_consensus.cc:2866] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:35.489758 13246 raft_consensus.cc:2258] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.487489 13245 raft_consensus.cc:374] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.489970 13245 raft_consensus.cc:729] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.490521 13245 consensus_queue.cc:260] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.490622 12922 leader_election.cc:302] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:35.491159 13434 raft_consensus.cc:2781] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:35.491256 13434 raft_consensus.cc:482] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:35.491356 13434 raft_consensus.cc:3037] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:35.491664 13245 ts_tablet_manager.cc:1398] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:35.494083 13434 raft_consensus.cc:504] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.494827 13245 tablet_bootstrap.cc:492] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.495846 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79486583025c42bfb1321f9cb6dd96f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:35.496039 13123 raft_consensus.cc:3037] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:35.496490 13245 tablet_bootstrap.cc:654] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.497373 13242 raft_consensus.cc:2229] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.497578 13242 raft_consensus.cc:2258] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.498494 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79486583025c42bfb1321f9cb6dd96f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:35.498821 13123 raft_consensus.cc:2454] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:35.499089 12921 leader_election.cc:341] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.499876 12922 leader_election.cc:302] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:35.500161 13353 raft_consensus.cc:2781] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:35.500347 13353 raft_consensus.cc:686] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:35.500638 13353 consensus_queue.cc:237] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.500845 13242 ts_tablet_manager.cc:1875] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.500906 13434 leader_election.cc:288] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:35.501962 13244 tablet_bootstrap.cc:654] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.503926 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet a80834187f874661b611b43065eee943 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:35.505131 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d380bf30599d4130bbb80b4f26837779 consensus state has no leader
I20240506 19:56:35.505527 12516 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a2ff5fed57d64244894f4af1f6067e85
W20240506 19:56:35.505875 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2519c03738864f6ab5dcfab7bc3fc8aa consensus state has no leader
I20240506 19:56:35.507747 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe (DEFAULT_TABLE table=default.Table_38260663814f40939e221b4885ed23a3 [id=c50645b97f4045d59c800f5df93bdf7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.508157 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be929f0e72e4d22a9af2fe6b3072bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.508703 13246 ts_tablet_manager.cc:1875] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.508711 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe (DEFAULT_TABLE table=default.Table_38260663814f40939e221b4885ed23a3 [id=c50645b97f4045d59c800f5df93bdf7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.508944 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.509135 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be929f0e72e4d22a9af2fe6b3072bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.509388 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 796628a0859c4511a8cb35e495dc8886 (DEFAULT_TABLE table=default.Table_9ec35e469b0e4d8ca1557a818fd7bee7 [id=7aa2f61e032f4db29e4bbd9d7505a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.509780 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796628a0859c4511a8cb35e495dc8886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.510372 12816 tablet_service.cc:1467] Processing CreateTablet for tablet f4e4be565e2d4c59b3587baed72e4a9c (DEFAULT_TABLE table=default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.510859 12822 tablet_service.cc:1467] Processing CreateTablet for tablet db13b58d17c94e0f82052f823915d9af (DEFAULT_TABLE table=default.Table_9321b5f2115c4ff8aa68a30db878b09c [id=21f83b4858344bf9959aee15e02a7bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.511719 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 796628a0859c4511a8cb35e495dc8886 (DEFAULT_TABLE table=default.Table_9ec35e469b0e4d8ca1557a818fd7bee7 [id=7aa2f61e032f4db29e4bbd9d7505a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.512118 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796628a0859c4511a8cb35e495dc8886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.516571 13104 tablet_service.cc:1467] Processing CreateTablet for tablet db13b58d17c94e0f82052f823915d9af (DEFAULT_TABLE table=default.Table_9321b5f2115c4ff8aa68a30db878b09c [id=21f83b4858344bf9959aee15e02a7bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.516729 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4be565e2d4c59b3587baed72e4a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.516927 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13b58d17c94e0f82052f823915d9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.517867 13249 tablet_bootstrap.cc:492] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.518016 13249 ts_tablet_manager.cc:1367] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.009s
I20240506 19:56:35.518846 13249 raft_consensus.cc:348] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.519061 13249 raft_consensus.cc:374] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.519153 13249 raft_consensus.cc:729] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.519340 13249 consensus_queue.cc:260] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.520184 13249 ts_tablet_manager.cc:1398] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:35.520357 13242 ts_tablet_manager.cc:1888] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.520467 13249 tablet_bootstrap.cc:492] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.520485 13242 log.cc:1196] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/7bdcdbb9cf3e4e9db6174bce7785565e
I20240506 19:56:35.520884 13242 ts_tablet_manager.cc:1909] T 7bdcdbb9cf3e4e9db6174bce7785565e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.521466 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13b58d17c94e0f82052f823915d9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.522145 13249 tablet_bootstrap.cc:654] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.524896 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91888e88234cbdbd00f48dc1574653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.525424 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.525839 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b with delete_type TABLET_DATA_DELETED (replaced by tablet 959f9e652bdd430898c1d3a95fe765f6) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.526083 13243 tablet_replica.cc:330] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:35.526400 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet f017e635010040008be72b3813b5a4ed with delete_type TABLET_DATA_DELETED (replaced by tablet 4a080ccac6394210b0b24b65843f9d75) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.526429 13243 raft_consensus.cc:2229] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.526633 13243 raft_consensus.cc:2258] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.526857 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.527369 13243 ts_tablet_manager.cc:1875] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.527390 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.527964 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe (DEFAULT_TABLE table=default.Table_38260663814f40939e221b4885ed23a3 [id=c50645b97f4045d59c800f5df93bdf7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.528383 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be929f0e72e4d22a9af2fe6b3072bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.530061 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 (DEFAULT_TABLE table=default.Table_69a8d24535d24338a12209fd418f1e1b [id=4021900d99ce4302b2de9b33a687e94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.530807 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.539878 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 7bdcdbb9cf3e4e9db6174bce7785565e (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:35.539796 13091 tablet_service.cc:1467] Processing CreateTablet for tablet f4e4be565e2d4c59b3587baed72e4a9c (DEFAULT_TABLE table=default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.540083 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_df1733520c4a4ae6a99a78f194d6cf06" } modify_external_catalogs: true
I20240506 19:56:35.540335 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4be565e2d4c59b3587baed72e4a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.540426 13242 tablet_replica.cc:330] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.540366 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_df1733520c4a4ae6a99a78f194d6cf06" } modify_external_catalogs: true
I20240506 19:56:35.542294 13242 raft_consensus.cc:2229] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.542866 13242 raft_consensus.cc:2258] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.544250 13242 ts_tablet_manager.cc:1875] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.551373 13244 tablet_bootstrap.cc:492] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.551918 13244 ts_tablet_manager.cc:1367] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.006s
I20240506 19:56:35.552011 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 (DEFAULT_TABLE table=default.Table_69a8d24535d24338a12209fd418f1e1b [id=4021900d99ce4302b2de9b33a687e94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.554088 12958 tablet_service.cc:1467] Processing CreateTablet for tablet db13b58d17c94e0f82052f823915d9af (DEFAULT_TABLE table=default.Table_9321b5f2115c4ff8aa68a30db878b09c [id=21f83b4858344bf9959aee15e02a7bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.554068 13244 raft_consensus.cc:348] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.554525 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db13b58d17c94e0f82052f823915d9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.556926 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79486583025c42bfb1321f9cb6dd96f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:35.557475 12921 leader_election.cc:341] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.561631 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 796628a0859c4511a8cb35e495dc8886 (DEFAULT_TABLE table=default.Table_9ec35e469b0e4d8ca1557a818fd7bee7 [id=7aa2f61e032f4db29e4bbd9d7505a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.562119 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796628a0859c4511a8cb35e495dc8886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.566569 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.568169 13244 raft_consensus.cc:374] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.568292 13244 raft_consensus.cc:729] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.569597 13245 tablet_bootstrap.cc:492] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.569783 13245 ts_tablet_manager.cc:1367] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.013s
I20240506 19:56:35.570919 13245 raft_consensus.cc:348] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.571417 13245 raft_consensus.cc:374] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.571498 13245 raft_consensus.cc:729] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.573752 13242 ts_tablet_manager.cc:1888] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.573896 13242 log.cc:1196] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d2abecd9ec7e4606b38c7d22477ddf65
I20240506 19:56:35.574245 13242 ts_tablet_manager.cc:1909] T d2abecd9ec7e4606b38c7d22477ddf65 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.575333 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d2abecd9ec7e4606b38c7d22477ddf65 (table default.Table_eb361931fc51481590136c98bc041bc6 [id=031d0b422342478ba95396f085780f3b]) successfully deleted
I20240506 19:56:35.572886 13245 consensus_queue.cc:260] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.575711 13244 consensus_queue.cc:260] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.576335 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1b7f56fea34946e590dae0621b28f9c6
I20240506 19:56:35.576891 12519 catalog_manager.cc:6554] ReplaceTablet: tablet 1b7f56fea34946e590dae0621b28f9c6 deleted and replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:35.577092 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.577307 12515 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet c0d7117c8dbf4c9f9faf295a7224fde6
I20240506 19:56:35.577922 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5516e8aad4e8474692b5b6c40bc483c1
I20240506 19:56:35.578413 12514 catalog_manager.cc:6554] ReplaceTablet: tablet 5516e8aad4e8474692b5b6c40bc483c1 deleted and replaced by tablet da41ef03182445f1946584c2bf015139
I20240506 19:56:35.579607 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 with delete_type TABLET_DATA_DELETED (replaced by tablet da41ef03182445f1946584c2bf015139) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.580149 13244 ts_tablet_manager.cc:1398] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240506 19:56:35.581143 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 (DEFAULT_TABLE table=default.Table_69a8d24535d24338a12209fd418f1e1b [id=4021900d99ce4302b2de9b33a687e94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.581627 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.582384 13245 ts_tablet_manager.cc:1398] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240506 19:56:35.582729 13245 tablet_bootstrap.cc:492] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.585072 13245 tablet_bootstrap.cc:654] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.585191 13244 tablet_bootstrap.cc:492] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.587257 13244 tablet_bootstrap.cc:654] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.589058 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:35.589211 12958 tablet_service.cc:1467] Processing CreateTablet for tablet f4e4be565e2d4c59b3587baed72e4a9c (DEFAULT_TABLE table=default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.589679 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4be565e2d4c59b3587baed72e4a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.590140 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8335f9308046279a25a4a95d8e0f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.593614 13242 tablet_replica.cc:330] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:35.593596 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4f91888e88234cbdbd00f48dc1574653 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f91888e88234cbdbd00f48dc1574653 already in progress: creating tablet
I20240506 19:56:35.593883 13242 raft_consensus.cc:2229] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:35.594133 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 72a3bed0eb234e21b890242e9d23a5b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a3bed0eb234e21b890242e9d23a5b4 already in progress: creating tablet
I20240506 19:56:35.594333 13242 raft_consensus.cc:2258] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.594759 13353 raft_consensus.cc:482] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.594810 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 with delete_type TABLET_DATA_DELETED (replaced by tablet da41ef03182445f1946584c2bf015139) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.594934 13353 raft_consensus.cc:504] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.595172 13242 ts_tablet_manager.cc:1875] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.595249 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8335f9308046279a25a4a95d8e0f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.595525 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.595685 13353 leader_election.cc:288] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.597242 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e3260f46024386baac64cb8677883c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:35.597441 13243 ts_tablet_manager.cc:1888] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.597575 13243 log.cc:1196] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b25ba6b9b24a48afb1c6f2ba8d6fd42b
W20240506 19:56:35.598012 12922 leader_election.cc:341] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.598003 13243 ts_tablet_manager.cc:1909] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:35.598675 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e3260f46024386baac64cb8677883c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:35.599174 13243 tablet_replica.cc:330] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:35.599373 13243 raft_consensus.cc:2229] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:35.599467 12921 leader_election.cc:341] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.599632 12921 leader_election.cc:302] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.599895 13353 raft_consensus.cc:2726] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:35.600301 13243 raft_consensus.cc:2258] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.600270 13329 raft_consensus.cc:482] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.600443 13329 raft_consensus.cc:504] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.600968 13243 ts_tablet_manager.cc:1875] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.601153 13329 leader_election.cc:288] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:35.601902 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e (DEFAULT_TABLE table=default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 [id=4b18d70e1c8847de81b7742f55f56260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.602339 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c09e3bf7046ccba6ed25b08e3460e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.604952 13246 ts_tablet_manager.cc:1888] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240506 19:56:35.605005 13057 leader_election.cc:341] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 18920e08b0de4f5292abb54e3432ee39
I20240506 19:56:35.605146 12825 tablet_service.cc:1467] Processing CreateTablet for tablet d6e54543d1544b1c8cff3e890e787337 (DEFAULT_TABLE table=default.Table_8b16088d777a4dcfb6b6d4be569babec [id=3045c42f00ac4ff19e0354945092c6d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.605566 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e54543d1544b1c8cff3e890e787337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.605762 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b (table default.Table_71fac0ef50de48c096c6328f3f213053 [id=a60eb0d8de524b8aa111777bca63ca8a]) successfully deleted
I20240506 19:56:35.605386 13246 log.cc:1196] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b25ba6b9b24a48afb1c6f2ba8d6fd42b
I20240506 19:56:35.606236 13246 ts_tablet_manager.cc:1909] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.607470 13246 tablet_replica.cc:330] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.607671 13246 raft_consensus.cc:2229] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.607854 13246 raft_consensus.cc:2258] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.608419 13246 ts_tablet_manager.cc:1875] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.609211 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:35.613811 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:35.613947 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 1372be71e1d141ed89e58cdb806431a4 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.614363 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1372be71e1d141ed89e58cdb806431a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.615546 13313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:35.615890 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18920e08b0de4f5292abb54e3432ee39" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:35.616838 13058 leader_election.cc:341] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:35.617018 13058 leader_election.cc:302] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:35.617226 13058 raft_consensus.cc:2670] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:35.617679 12816 tablet_service.cc:1467] Processing CreateTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 (DEFAULT_TABLE table=default.Table_6da5f3268c67468f810ffaaa56d3cfdd [id=8180a10700c54e20b558796b518d74b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.618279 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef03fcaa05d04ebd9f262f43ff7c02e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.618501 12962 tablet_service.cc:1467] Processing CreateTablet for tablet d6e54543d1544b1c8cff3e890e787337 (DEFAULT_TABLE table=default.Table_8b16088d777a4dcfb6b6d4be569babec [id=3045c42f00ac4ff19e0354945092c6d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.618918 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e54543d1544b1c8cff3e890e787337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.619330 13244 tablet_bootstrap.cc:492] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.619509 13244 ts_tablet_manager.cc:1367] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240506 19:56:35.620455 13244 raft_consensus.cc:348] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.620599 13245 tablet_bootstrap.cc:492] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.620666 13244 raft_consensus.cc:374] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.620743 13245 ts_tablet_manager.cc:1367] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.008s
I20240506 19:56:35.620757 13244 raft_consensus.cc:729] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.621004 13244 consensus_queue.cc:260] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.621605 13244 ts_tablet_manager.cc:1398] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:35.621484 13245 raft_consensus.cc:348] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.621698 13245 raft_consensus.cc:374] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.621788 13245 raft_consensus.cc:729] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.622033 13245 consensus_queue.cc:260] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.624820 12956 tablet_service.cc:1467] Processing CreateTablet for tablet e17197adc6ca4593ac3fc494548a6e67 (DEFAULT_TABLE table=default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.625322 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17197adc6ca4593ac3fc494548a6e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.626104 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 1372be71e1d141ed89e58cdb806431a4 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.626616 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1372be71e1d141ed89e58cdb806431a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.627647 13244 tablet_bootstrap.cc:492] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.628216 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2c9f4baf55412d9bc00818a00c652d (DEFAULT_TABLE table=default.Table_b53ac51ea36943aa94434bfe07d4dfa5 [id=034d7824127b4f94aa24bdeb76bfbd7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.628679 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2c9f4baf55412d9bc00818a00c652d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.631269 13244 tablet_bootstrap.cc:654] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.632072 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e (DEFAULT_TABLE table=default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 [id=4b18d70e1c8847de81b7742f55f56260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.632613 12819 tablet_service.cc:1467] Processing CreateTablet for tablet e17197adc6ca4593ac3fc494548a6e67 (DEFAULT_TABLE table=default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.633033 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c09e3bf7046ccba6ed25b08e3460e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.633038 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17197adc6ca4593ac3fc494548a6e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.634207 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2c9f4baf55412d9bc00818a00c652d (DEFAULT_TABLE table=default.Table_b53ac51ea36943aa94434bfe07d4dfa5 [id=034d7824127b4f94aa24bdeb76bfbd7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.634582 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2c9f4baf55412d9bc00818a00c652d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.636082 13245 ts_tablet_manager.cc:1398] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240506 19:56:35.641014 13313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20240506 19:56:35.641016 12508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.253:36819, user_credentials={real_user=slave}} blocked reactor thread for 101699us
I20240506 19:56:35.642174 13245 tablet_bootstrap.cc:492] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.642244 13249 tablet_bootstrap.cc:492] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.642416 13249 ts_tablet_manager.cc:1367] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.122s user 0.006s sys 0.001s
I20240506 19:56:35.642541 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.643347 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 with delete_type TABLET_DATA_DELETED (replaced by tablet da41ef03182445f1946584c2bf015139) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.643429 13249 raft_consensus.cc:348] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.643692 13249 raft_consensus.cc:374] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.643790 13249 raft_consensus.cc:729] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.644124 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8335f9308046279a25a4a95d8e0f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.644124 13249 consensus_queue.cc:260] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.644706 13313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:35.644811 13249 ts_tablet_manager.cc:1398] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:35.645139 13249 tablet_bootstrap.cc:492] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.644922 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e (DEFAULT_TABLE table=default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 [id=4b18d70e1c8847de81b7742f55f56260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.645628 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c09e3bf7046ccba6ed25b08e3460e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.645136 13242 ts_tablet_manager.cc:1888] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.647063 13242 log.cc:1196] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/18920e08b0de4f5292abb54e3432ee39
I20240506 19:56:35.648581 13245 tablet_bootstrap.cc:654] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.648977 12958 tablet_service.cc:1467] Processing CreateTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 (DEFAULT_TABLE table=default.Table_6da5f3268c67468f810ffaaa56d3cfdd [id=8180a10700c54e20b558796b518d74b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.649477 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef03fcaa05d04ebd9f262f43ff7c02e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.649500 13242 ts_tablet_manager.cc:1909] T 18920e08b0de4f5292abb54e3432ee39 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.652035 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 18920e08b0de4f5292abb54e3432ee39 (table default.Table_fcd8a4763edf472692c23bb235cb7852 [id=229d13dce55e4738973b0180ddf23c5e]) successfully deleted
I20240506 19:56:35.653012 13242 tablet_replica.cc:330] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.653199 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 1372be71e1d141ed89e58cdb806431a4 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.653314 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.653662 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1372be71e1d141ed89e58cdb806431a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.653750 13242 raft_consensus.cc:2229] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.653807 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.653919 13242 raft_consensus.cc:2258] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.654271 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 8596757ef75e433fb801f7679db5fefb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.654945 13091 tablet_service.cc:1467] Processing CreateTablet for tablet d6e54543d1544b1c8cff3e890e787337 (DEFAULT_TABLE table=default.Table_8b16088d777a4dcfb6b6d4be569babec [id=3045c42f00ac4ff19e0354945092c6d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.655332 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e54543d1544b1c8cff3e890e787337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.655659 13242 ts_tablet_manager.cc:1875] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.658928 13094 tablet_service.cc:1467] Processing CreateTablet for tablet e17197adc6ca4593ac3fc494548a6e67 (DEFAULT_TABLE table=default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.659966 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2c9f4baf55412d9bc00818a00c652d (DEFAULT_TABLE table=default.Table_b53ac51ea36943aa94434bfe07d4dfa5 [id=034d7824127b4f94aa24bdeb76bfbd7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.660085 13249 tablet_bootstrap.cc:654] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.660259 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17197adc6ca4593ac3fc494548a6e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.662786 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2c9f4baf55412d9bc00818a00c652d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.664465 13353 raft_consensus.cc:482] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.664659 13353 raft_consensus.cc:504] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.665503 13353 leader_election.cc:288] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.666169 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16704567ef6454eaef886fb9cd599ce" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:35.666858 13244 tablet_bootstrap.cc:492] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:56:35.666862 12921 leader_election.cc:341] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.667035 13244 ts_tablet_manager.cc:1367] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.007s
I20240506 19:56:35.667223 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b (table default.Table_71fac0ef50de48c096c6328f3f213053 [id=a60eb0d8de524b8aa111777bca63ca8a]) successfully deleted
I20240506 19:56:35.668022 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_915828e8f55f4e1687a45a7fe7fa4f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.668103 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3a6afcad38024709a1f104fa5bbadcff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.668666 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_915828e8f55f4e1687a45a7fe7fa4f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.668421 13244 raft_consensus.cc:348] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:35.668862 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a6afcad38024709a1f104fa5bbadcff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.668912 13244 raft_consensus.cc:374] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.668995 13244 raft_consensus.cc:729] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.669246 13244 consensus_queue.cc:260] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.671478 13244 ts_tablet_manager.cc:1398] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:35.671790 13244 tablet_bootstrap.cc:492] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.667554 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16704567ef6454eaef886fb9cd599ce" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:35.672885 12922 leader_election.cc:341] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.673027 12922 leader_election.cc:302] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.673286 13243 ts_tablet_manager.cc:1888] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.673444 13243 log.cc:1196] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f017e635010040008be72b3813b5a4ed
I20240506 19:56:35.673854 13243 ts_tablet_manager.cc:1909] T f017e635010040008be72b3813b5a4ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:35.674688 13244 tablet_bootstrap.cc:654] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.674870 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f017e635010040008be72b3813b5a4ed (table default.Table_bb529dd8d37f40449f27b0e07cbc93dc [id=4b9a2f4ea4b1460f89e8981e952e0d2d]) successfully deleted
I20240506 19:56:35.676113 13243 tablet_replica.cc:330] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:35.676337 13243 raft_consensus.cc:2229] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.676522 13243 raft_consensus.cc:2258] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.676952 13353 raft_consensus.cc:2726] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:35.677060 13243 ts_tablet_manager.cc:1875] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.681789 13242 ts_tablet_manager.cc:1888] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.682163 13242 log.cc:1196] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b5adab0af6274ba0ad54402c5b1ff757
I20240506 19:56:35.683138 13242 ts_tablet_manager.cc:1909] T b5adab0af6274ba0ad54402c5b1ff757 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.684016 13244 tablet_bootstrap.cc:492] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.684937 13244 ts_tablet_manager.cc:1367] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240506 19:56:35.685832 13244 raft_consensus.cc:348] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.686142 13244 raft_consensus.cc:374] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.686300 13244 raft_consensus.cc:729] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.686643 13244 consensus_queue.cc:260] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.689615 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c57d5247a9fc4628ac83a4fd916f2b53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.690718 13243 ts_tablet_manager.cc:1888] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.690881 13243 log.cc:1196] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/44de7153689b4dbd80dae87d3ee48a83
I20240506 19:56:35.691270 13244 ts_tablet_manager.cc:1398] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:35.691278 13243 ts_tablet_manager.cc:1909] T 44de7153689b4dbd80dae87d3ee48a83 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:35.691414 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 2502b423bd1040539fb1a3ebbece3c1c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.692335 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d8a9ecf47d7474c82758417449d940c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.692365 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 44de7153689b4dbd80dae87d3ee48a83 (table default.Table_76747eb355e74563ad356ac0b78eeb8e [id=4ef402284c534d928551d10f8857c666]) successfully deleted
I20240506 19:56:35.693255 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b4ce4fd613b049df9285a95fe23671c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.693131 13092 tablet_service.cc:1467] Processing CreateTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 (DEFAULT_TABLE table=default.Table_6da5f3268c67468f810ffaaa56d3cfdd [id=8180a10700c54e20b558796b518d74b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.693621 13245 tablet_bootstrap.cc:492] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.694382 13245 ts_tablet_manager.cc:1367] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.003s
I20240506 19:56:35.694109 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef03fcaa05d04ebd9f262f43ff7c02e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.695592 13244 tablet_bootstrap.cc:492] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.697283 13244 tablet_bootstrap.cc:654] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.698225 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b058641e17b2474785e2066bd9781690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.698356 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1a0e2e4b842f4375ab63b84f691c796b" } modify_external_catalogs: true
I20240506 19:56:35.698737 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1a0e2e4b842f4375ab63b84f691c796b" } modify_external_catalogs: true
I20240506 19:56:35.699831 13246 ts_tablet_manager.cc:1888] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.700066 13246 log.cc:1196] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/18920e08b0de4f5292abb54e3432ee39
I20240506 19:56:35.700534 13246 ts_tablet_manager.cc:1909] T 18920e08b0de4f5292abb54e3432ee39 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.701440 13244 tablet_bootstrap.cc:492] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.701622 13244 ts_tablet_manager.cc:1367] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240506 19:56:35.702195 13246 tablet_replica.cc:330] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.702371 13244 raft_consensus.cc:348] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.702504 13246 raft_consensus.cc:2229] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.702641 13244 raft_consensus.cc:374] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.702733 13244 raft_consensus.cc:729] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.702849 13246 raft_consensus.cc:2258] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.704937 13244 consensus_queue.cc:260] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.706343 13249 tablet_bootstrap.cc:492] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.706607 13249 ts_tablet_manager.cc:1367] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.005s
I20240506 19:56:35.707679 13246 ts_tablet_manager.cc:1875] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.708508 13249 raft_consensus.cc:348] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.708855 13249 raft_consensus.cc:374] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.709146 13249 raft_consensus.cc:729] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.709741 13249 consensus_queue.cc:260] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.711418 13249 ts_tablet_manager.cc:1398] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240506 19:56:35.711510 13244 ts_tablet_manager.cc:1398] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:35.712153 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d91e484ef0cc4adb86af57d549a7e641" } modify_external_catalogs: true
I20240506 19:56:35.712376 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d91e484ef0cc4adb86af57d549a7e641" } modify_external_catalogs: true
I20240506 19:56:35.713027 13245 raft_consensus.cc:348] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.714076 13245 raft_consensus.cc:374] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.714395 13245 raft_consensus.cc:729] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.715041 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9321b5f2115c4ff8aa68a30db878b09c" } new_table_name: "default.Table_e882abe9665a4511bd29de22b6256710" modify_external_catalogs: true
I20240506 19:56:35.715068 13244 tablet_bootstrap.cc:492] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.687813 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b5adab0af6274ba0ad54402c5b1ff757 (table default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 [id=48be0d45bbf34e6187819fd3994a0afe]) successfully deleted
W20240506 19:56:35.716785 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1b7f56fea34946e590dae0621b28f9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b7f56fea34946e590dae0621b28f9c6 already in progress: creating tablet
W20240506 19:56:35.717044 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5516e8aad4e8474692b5b6c40bc483c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5516e8aad4e8474692b5b6c40bc483c1 already in progress: creating tablet
W20240506 19:56:35.717267 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet cc8335f9308046279a25a4a95d8e0f65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc8335f9308046279a25a4a95d8e0f65 already in progress: creating tablet
I20240506 19:56:35.717422 13243 tablet_replica.cc:330] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:35.717443 13244 tablet_bootstrap.cc:654] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.718155 13243 raft_consensus.cc:2229] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.718376 13243 raft_consensus.cc:2258] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.718935 13243 ts_tablet_manager.cc:1875] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.720594 13249 tablet_bootstrap.cc:492] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.722234 13249 tablet_bootstrap.cc:654] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.721908 13245 consensus_queue.cc:260] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.729966 13249 tablet_bootstrap.cc:492] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
W20240506 19:56:35.730090 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db13b58d17c94e0f82052f823915d9af consensus state has no leader
I20240506 19:56:35.729830 13245 ts_tablet_manager.cc:1398] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.035s user 0.004s sys 0.000s
I20240506 19:56:35.730183 13249 ts_tablet_manager.cc:1367] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240506 19:56:35.731108 13249 raft_consensus.cc:348] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.731388 13249 raft_consensus.cc:374] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.731467 13249 raft_consensus.cc:729] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.731688 13249 consensus_queue.cc:260] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.732820 13249 ts_tablet_manager.cc:1398] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 19:56:35.733175 13249 tablet_bootstrap.cc:492] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.734964 13245 tablet_bootstrap.cc:492] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.738932 13249 tablet_bootstrap.cc:654] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.740325 13243 ts_tablet_manager.cc:1888] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.740463 13243 log.cc:1196] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b5adab0af6274ba0ad54402c5b1ff757
I20240506 19:56:35.740823 13243 ts_tablet_manager.cc:1909] T b5adab0af6274ba0ad54402c5b1ff757 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:35.741902 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b5adab0af6274ba0ad54402c5b1ff757 (table default.Table_4859ef0b7bf94e8fb8d08ed18049a2e9 [id=48be0d45bbf34e6187819fd3994a0afe]) successfully deleted
W20240506 19:56:35.746645 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4ce4fd613b049df9285a95fe23671c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.747431 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5516e8aad4e8474692b5b6c40bc483c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5516e8aad4e8474692b5b6c40bc483c1 already in progress: creating tablet
W20240506 19:56:35.746675 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b058641e17b2474785e2066bd9781690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.748185 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet cc8335f9308046279a25a4a95d8e0f65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc8335f9308046279a25a4a95d8e0f65 already in progress: creating tablet
W20240506 19:56:35.748824 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1b7f56fea34946e590dae0621b28f9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b7f56fea34946e590dae0621b28f9c6 already in progress: creating tablet
W20240506 19:56:35.749827 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c57d5247a9fc4628ac83a4fd916f2b53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.752951 13249 tablet_bootstrap.cc:492] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.753327 13249 ts_tablet_manager.cc:1367] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240506 19:56:35.753791 13245 tablet_bootstrap.cc:654] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.754213 13249 raft_consensus.cc:348] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.754426 13249 raft_consensus.cc:374] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.754510 13249 raft_consensus.cc:729] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.754768 13249 consensus_queue.cc:260] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.755326 13249 ts_tablet_manager.cc:1398] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:35.755694 13249 tablet_bootstrap.cc:492] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.757449 13249 tablet_bootstrap.cc:654] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.763180 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9d7abbf595704ae18e43dc8996a642aa" } new_table_name: "default.Table_ffcc8db8756e4baf9a6595678ba1b3eb" modify_external_catalogs: true
I20240506 19:56:35.763448 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_69a8d24535d24338a12209fd418f1e1b" } new_table_name: "default.Table_0f2e3be5e4274da5ac5270fced5e3d84" modify_external_catalogs: true
I20240506 19:56:35.763396 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b53ae7ee595f4d70813b67094f238eca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.763630 13244 tablet_bootstrap.cc:492] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.763800 13244 ts_tablet_manager.cc:1367] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.009s
W20240506 19:56:35.764039 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b53ae7ee595f4d70813b67094f238eca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.764583 13244 raft_consensus.cc:348] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.764798 13244 raft_consensus.cc:374] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.764894 13244 raft_consensus.cc:729] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.765179 13244 consensus_queue.cc:260] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.765544 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6da5f3268c67468f810ffaaa56d3cfdd" } new_table_name: "default.Table_2cbe1f98afef411bbef910c6931cabd1" modify_external_catalogs: true
I20240506 19:56:35.766084 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d5bb6a8c548e46ecb59035f49816ad52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.766981 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5bb6a8c548e46ecb59035f49816ad52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.767251 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.768225 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 18920e08b0de4f5292abb54e3432ee39 (table default.Table_fcd8a4763edf472692c23bb235cb7852 [id=229d13dce55e4738973b0180ddf23c5e]) successfully deleted
I20240506 19:56:35.769048 13244 ts_tablet_manager.cc:1398] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:35.772874 12519 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]). Expected version 3 got 0
I20240506 19:56:35.774021 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f4146ffd1ac744c4bdd3506d398dcdc0" } modify_external_catalogs: true
I20240506 19:56:35.774317 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f4146ffd1ac744c4bdd3506d398dcdc0" } modify_external_catalogs: true
I20240506 19:56:35.775607 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b53ac51ea36943aa94434bfe07d4dfa5" } new_table_name: "default.Table_adbb55d8d99f4266a048d9a935d73195" modify_external_catalogs: true
I20240506 19:56:35.794296 13249 tablet_bootstrap.cc:492] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.794488 13249 ts_tablet_manager.cc:1367] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.001s
I20240506 19:56:35.795233 13249 raft_consensus.cc:348] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.795472 13249 raft_consensus.cc:374] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.795545 13249 raft_consensus.cc:729] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.795814 13249 consensus_queue.cc:260] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.796332 13249 ts_tablet_manager.cc:1398] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:35.796690 13249 tablet_bootstrap.cc:492] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.797924 13353 raft_consensus.cc:482] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.798074 13353 raft_consensus.cc:504] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.798485 13249 tablet_bootstrap.cc:654] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.798800 13353 leader_election.cc:288] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.799712 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e714ed6b64f2d84770e7182e4103a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:35.800719 12922 leader_election.cc:341] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.802505 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e714ed6b64f2d84770e7182e4103a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:35.803390 12921 leader_election.cc:341] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.803663 12921 leader_election.cc:302] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:35.804594 13353 raft_consensus.cc:2726] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:35.805757 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41faccd54f3e49ddb48ea5234f6cdab5 consensus state has no leader
I20240506 19:56:35.806663 13244 tablet_bootstrap.cc:492] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:35.807401 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 consensus state has no leader
I20240506 19:56:35.809522 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet cc8335f9308046279a25a4a95d8e0f65 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:35.811828 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef03fcaa05d04ebd9f262f43ff7c02e4 consensus state has no leader
I20240506 19:56:35.811913 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet e34e128329204418b7d6fb55762fb590 (table default.Table_f44dcd3362d64342ad02f7d17b3f24d0 [id=a147e46c173c402db02793b1e11de5b5]). Expected version 1 got 0
I20240506 19:56:35.812273 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_62b69a40ec6a432186ff79fc5693659e [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 5 got 0
I20240506 19:56:35.816527 13104 tablet_service.cc:1467] Processing CreateTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 (DEFAULT_TABLE table=default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.817004 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b3ed0b56aa478abb44347afca8fcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.821094 13468 tablet.cc:1722] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:35.821715 12958 tablet_service.cc:1467] Processing CreateTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 (DEFAULT_TABLE table=default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.822103 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b3ed0b56aa478abb44347afca8fcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.822033 13470 tablet.cc:1722] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240506 19:56:35.822459 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e2c9f4baf55412d9bc00818a00c652d consensus state has no leader
I20240506 19:56:35.825259 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet e34e128329204418b7d6fb55762fb590 (table default.Table_f44dcd3362d64342ad02f7d17b3f24d0 [id=a147e46c173c402db02793b1e11de5b5]). Expected version 1 got 0
I20240506 19:56:35.825598 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 88dba275893f4696a5318ec21f0b2477 (DEFAULT_TABLE table=default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 [id=194197559e4e4469b5a3622465e90f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.825965 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88dba275893f4696a5318ec21f0b2477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.809274 13244 tablet_bootstrap.cc:654] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.814580 12816 tablet_service.cc:1467] Processing CreateTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 (DEFAULT_TABLE table=default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.827175 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b3ed0b56aa478abb44347afca8fcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.827808 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2f78de17b14b4a3c85876eb23bee958a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.828426 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f78de17b14b4a3c85876eb23bee958a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.829792 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 (DEFAULT_TABLE table=default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.830224 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d23f0a5ff6d430282256e9bddc186b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.814846 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 88dba275893f4696a5318ec21f0b2477 (DEFAULT_TABLE table=default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 [id=194197559e4e4469b5a3622465e90f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.831172 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88dba275893f4696a5318ec21f0b2477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.831473 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b with delete_type TABLET_DATA_DELETED (replaced by tablet 959f9e652bdd430898c1d3a95fe765f6) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.831897 13242 tablet_replica.cc:330] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.832154 13242 raft_consensus.cc:2229] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.832374 13242 raft_consensus.cc:2258] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.833513 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.833546 13242 ts_tablet_manager.cc:1875] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.814690 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 (DEFAULT_TABLE table=default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.834437 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 88dba275893f4696a5318ec21f0b2477 (DEFAULT_TABLE table=default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 [id=194197559e4e4469b5a3622465e90f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.834569 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet f017e635010040008be72b3813b5a4ed with delete_type TABLET_DATA_DELETED (replaced by tablet 4a080ccac6394210b0b24b65843f9d75) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.834880 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88dba275893f4696a5318ec21f0b2477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.834928 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de7153689b4dbd80dae87d3ee48a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.836905 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d23f0a5ff6d430282256e9bddc186b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.840826 12955 tablet_service.cc:1467] Processing CreateTablet for tablet da41ef03182445f1946584c2bf015139 (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.841444 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 104337875517445f862d82ece3a84889 (DEFAULT_TABLE table=default.Table_4b69070c2c254336bfaec6ea640736d0 [id=54457b4758cf4a5b8f7a55ef6b21f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.843148 13100 tablet_service.cc:1467] Processing CreateTablet for tablet da41ef03182445f1946584c2bf015139 (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.843556 13245 tablet_bootstrap.cc:492] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.843693 13245 ts_tablet_manager.cc:1367] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.110s user 0.002s sys 0.002s
I20240506 19:56:35.843792 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 104337875517445f862d82ece3a84889 (DEFAULT_TABLE table=default.Table_4b69070c2c254336bfaec6ea640736d0 [id=54457b4758cf4a5b8f7a55ef6b21f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.844302 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da41ef03182445f1946584c2bf015139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.845074 13246 ts_tablet_manager.cc:1888] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.845283 13246 log.cc:1196] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/09856d2dd66043fbab08966d04e7b992
I20240506 19:56:35.846592 13246 ts_tablet_manager.cc:1909] T 09856d2dd66043fbab08966d04e7b992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.847154 13245 raft_consensus.cc:348] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.847389 13245 raft_consensus.cc:374] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.847493 13245 raft_consensus.cc:729] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.847715 13245 consensus_queue.cc:260] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.848105 13246 tablet_replica.cc:330] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:35.848351 13245 ts_tablet_manager.cc:1398] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240506 19:56:35.849457 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104337875517445f862d82ece3a84889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.855441 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 09856d2dd66043fbab08966d04e7b992 (table default.Table_0f2e3be5e4274da5ac5270fced5e3d84 [id=4021900d99ce4302b2de9b33a687e94f]) successfully deleted
I20240506 19:56:35.856096 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0e976f7b85cd40e49f346f31b8a5d1ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.856822 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e976f7b85cd40e49f346f31b8a5d1ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.857342 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_577b201c2bf345bdbaeb7bda56b8296c" } modify_external_catalogs: true
I20240506 19:56:35.857570 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_577b201c2bf345bdbaeb7bda56b8296c" } modify_external_catalogs: true
I20240506 19:56:35.858551 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f44dcd3362d64342ad02f7d17b3f24d0" } modify_external_catalogs: true
I20240506 19:56:35.858757 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f44dcd3362d64342ad02f7d17b3f24d0" } modify_external_catalogs: true
I20240506 19:56:35.864406 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8ae600b2261b45e8ba254d1767420f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.864745 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da41ef03182445f1946584c2bf015139. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:35.865031 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ae600b2261b45e8ba254d1767420f77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.865351 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104337875517445f862d82ece3a84889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.867439 13246 raft_consensus.cc:2229] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.868800 13246 raft_consensus.cc:2258] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.871436 13252 raft_consensus.cc:482] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.871627 13252 raft_consensus.cc:504] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.872350 13252 leader_election.cc:288] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:35.873262 13246 ts_tablet_manager.cc:1875] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:35.873510 12781 leader_election.cc:341] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f017e635010040008be72b3813b5a4ed
W20240506 19:56:35.874063 12783 leader_election.cc:341] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f017e635010040008be72b3813b5a4ed
I20240506 19:56:35.874218 12783 leader_election.cc:302] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:35.874857 13252 raft_consensus.cc:2726] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:35.881930 13245 tablet_bootstrap.cc:492] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.882642 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 104337875517445f862d82ece3a84889 (DEFAULT_TABLE table=default.Table_4b69070c2c254336bfaec6ea640736d0 [id=54457b4758cf4a5b8f7a55ef6b21f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.883113 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104337875517445f862d82ece3a84889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.884969 13245 tablet_bootstrap.cc:654] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.890441 12816 tablet_service.cc:1467] Processing CreateTablet for tablet da41ef03182445f1946584c2bf015139 (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.890939 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da41ef03182445f1946584c2bf015139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.896261 13242 ts_tablet_manager.cc:1888] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.896853 13242 log.cc:1196] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b25ba6b9b24a48afb1c6f2ba8d6fd42b
I20240506 19:56:35.897045 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 (DEFAULT_TABLE table=default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.825731 13469 tablet.cc:1722] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:35.896523 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet b5adab0af6274ba0ad54402c5b1ff757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:32 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.897982 13242 ts_tablet_manager.cc:1909] T b25ba6b9b24a48afb1c6f2ba8d6fd42b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:35.898455 13249 tablet_bootstrap.cc:492] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.898633 13249 ts_tablet_manager.cc:1367] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.005s
I20240506 19:56:35.899498 13249 raft_consensus.cc:348] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.899729 13249 raft_consensus.cc:374] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.899835 13249 raft_consensus.cc:729] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.900130 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:35.900107 13249 consensus_queue.cc:260] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.900789 13249 ts_tablet_manager.cc:1398] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:35.900911 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 2751074a54ed4bcea6979e6ad11c2efe from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.901093 13249 tablet_bootstrap.cc:492] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:35.901527 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.902240 13242 tablet_replica.cc:330] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:35.902428 13242 raft_consensus.cc:2229] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.902598 13242 raft_consensus.cc:2258] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.903409 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:35.904250 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 20c1590e36b4471d8e72f4b07410ee0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20c1590e36b4471d8e72f4b07410ee0f already in progress: creating tablet
I20240506 19:56:35.904912 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b25ba6b9b24a48afb1c6f2ba8d6fd42b (table default.Table_71fac0ef50de48c096c6328f3f213053 [id=a60eb0d8de524b8aa111777bca63ca8a]) successfully deleted
I20240506 19:56:35.907466 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.909780 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d23f0a5ff6d430282256e9bddc186b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.909857 13246 ts_tablet_manager.cc:1888] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.910018 13246 log.cc:1196] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/30cb3cb7cbe14cc9bf1bfd07227f538e
I20240506 19:56:35.911687 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:35.912357 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 1667a0bbd3af4d678434b068c47cc518 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.912930 12518 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a] alter complete (version 3)
I20240506 19:56:35.914024 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e34e128329204418b7d6fb55762fb590
I20240506 19:56:35.914346 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 88dba275893f4696a5318ec21f0b2477
I20240506 19:56:35.915413 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.915473 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.915735 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8842523a97954e3f81ffd87530201d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.916759 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet e34e128329204418b7d6fb55762fb590 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.916993 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 88dba275893f4696a5318ec21f0b2477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.919730 13249 tablet_bootstrap.cc:654] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.921284 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 88dba275893f4696a5318ec21f0b2477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.921607 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet a16704567ef6454eaef886fb9cd599ce with delete_type TABLET_DATA_DELETED (replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.921741 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 79486583025c42bfb1321f9cb6dd96f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.921959 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.922156 13243 tablet_replica.cc:330] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:35.922214 13242 ts_tablet_manager.cc:1875] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.922363 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet e34e128329204418b7d6fb55762fb590 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.922389 13243 raft_consensus.cc:2229] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.922578 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet d843fe82af1148f4a9045ce8b09ad11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.922613 13243 raft_consensus.cc:2258] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.923151 13243 ts_tablet_manager.cc:1875] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.924723 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:35.924935 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]). Expected version 1 got 0
I20240506 19:56:35.925455 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 8596757ef75e433fb801f7679db5fefb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.925695 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.925897 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e3260f46024386baac64cb8677883c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.926231 13249 tablet_bootstrap.cc:492] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:35.926463 13249 ts_tablet_manager.cc:1367] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20240506 19:56:35.927665 13249 raft_consensus.cc:348] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.927913 13249 raft_consensus.cc:374] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.928009 13249 raft_consensus.cc:729] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.928334 13249 consensus_queue.cc:260] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:35.928886 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 88dba275893f4696a5318ec21f0b2477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88dba275893f4696a5318ec21f0b2477 already in progress: creating tablet
I20240506 19:56:35.929652 13244 tablet_bootstrap.cc:492] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:35.929845 13244 ts_tablet_manager.cc:1367] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.012s
I20240506 19:56:35.929994 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 88dba275893f4696a5318ec21f0b2477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.930006 13243 ts_tablet_manager.cc:1888] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.930223 13243 log.cc:1196] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:35.930719 13243 ts_tablet_manager.cc:1909] T a16704567ef6454eaef886fb9cd599ce P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:35.931221 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet e34e128329204418b7d6fb55762fb590 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.931490 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:35.931985 12513 catalog_manager.cc:6554] ReplaceTablet: tablet 2751074a54ed4bcea6979e6ad11c2efe deleted and replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7
I20240506 19:56:35.932399 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe with delete_type TABLET_DATA_DELETED (replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.933817 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe with delete_type TABLET_DATA_DELETED (replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.934518 13245 tablet_bootstrap.cc:492] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:35.934701 13245 ts_tablet_manager.cc:1367] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.003s
I20240506 19:56:35.934800 13244 raft_consensus.cc:348] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.935088 13244 raft_consensus.cc:374] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.935178 13244 raft_consensus.cc:729] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.935395 13245 raft_consensus.cc:348] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.935433 13244 consensus_queue.cc:260] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.935624 13245 raft_consensus.cc:374] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:35.935703 13245 raft_consensus.cc:729] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:35.935945 13245 consensus_queue.cc:260] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:35.936484 13245 ts_tablet_manager.cc:1398] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:35.937402 13245 tablet_bootstrap.cc:492] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:35.939163 13246 ts_tablet_manager.cc:1909] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:35.940727 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 30cb3cb7cbe14cc9bf1bfd07227f538e (table default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc [id=8b4499b016334533ab4366c2a4b1bd72]) successfully deleted
I20240506 19:56:35.941408 13245 tablet_bootstrap.cc:654] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:35.941911 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b38453bf678748399339806d12993c88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b38453bf678748399339806d12993c88 already in progress: creating tablet
I20240506 19:56:35.942040 13244 ts_tablet_manager.cc:1398] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
W20240506 19:56:35.942902 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d6c656e16c1045dea79bc23b64fdd5e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c656e16c1045dea79bc23b64fdd5e8 already in progress: creating tablet
I20240506 19:56:35.943231 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet d843fe82af1148f4a9045ce8b09ad11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:35.943614 13244 tablet_bootstrap.cc:492] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:35.945165 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 09856d2dd66043fbab08966d04e7b992 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d04a1c5ee7b4d529fa2f4da4945ff09) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:35.946171 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4f91888e88234cbdbd00f48dc1574653 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f91888e88234cbdbd00f48dc1574653 already in progress: creating tablet
I20240506 19:56:35.946588 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.946700 13246 tablet_replica.cc:330] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:35.946700 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 72a3bed0eb234e21b890242e9d23a5b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a3bed0eb234e21b890242e9d23a5b4 already in progress: creating tablet
I20240506 19:56:35.946907 13246 raft_consensus.cc:2229] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:35.946923 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1b7f56fea34946e590dae0621b28f9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b7f56fea34946e590dae0621b28f9c6 already in progress: creating tablet
I20240506 19:56:35.947116 13246 raft_consensus.cc:2258] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:35.947168 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5516e8aad4e8474692b5b6c40bc483c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5516e8aad4e8474692b5b6c40bc483c1 already in progress: creating tablet
I20240506 19:56:35.947714 13246 ts_tablet_manager.cc:1875] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.948431 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.948771 13242 ts_tablet_manager.cc:1888] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.949095 13434 raft_consensus.cc:482] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.949232 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.949395 13434 raft_consensus.cc:504] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.949759 13243 tablet_replica.cc:330] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:35.949947 13243 raft_consensus.cc:2229] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:35.950191 13243 pending_rounds.cc:62] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:35.950244 13434 leader_election.cc:288] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.951206 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34e128329204418b7d6fb55762fb590" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:35.947918 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet cc8335f9308046279a25a4a95d8e0f65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc8335f9308046279a25a4a95d8e0f65 already in progress: creating tablet
I20240506 19:56:35.952345 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a16704567ef6454eaef886fb9cd599ce (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
W20240506 19:56:35.952365 12922 leader_election.cc:341] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:35.950248 13243 pending_rounds.cc:68] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744020057198592 op_type: NO_OP noop_request { }
I20240506 19:56:35.952709 13243 raft_consensus.cc:2866] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:35.952838 13243 raft_consensus.cc:2258] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.953666 13243 ts_tablet_manager.cc:1875] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:35.957993 13433 raft_consensus.cc:482] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:35.958170 13433 raft_consensus.cc:504] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:35.958678 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:35.958950 13433 leader_election.cc:288] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:35.959903 13243 ts_tablet_manager.cc:1888] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:35.960057 13243 log.cc:1196] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/79486583025c42bfb1321f9cb6dd96f1
I20240506 19:56:35.960407 13243 ts_tablet_manager.cc:1909] T 79486583025c42bfb1321f9cb6dd96f1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:35.960561 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a223c33ba56c4ddbba46fedbce660ae2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:35.960799 13124 raft_consensus.cc:2454] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:35.961239 12922 leader_election.cc:302] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:35.961395 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 79486583025c42bfb1321f9cb6dd96f1 (table default.Table_5f69164a5a01441fa6dc72d34b878eb2 [id=bc2b2ac5e9704dd48c4cd3620e7074e3]) successfully deleted
I20240506 19:56:35.961634 13243 tablet_replica.cc:330] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:35.961720 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 40ece0ad4df247eb9d380ed1c755ac99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40ece0ad4df247eb9d380ed1c755ac99 already in progress: creating tablet
I20240506 19:56:35.961723 13433 raft_consensus.cc:2781] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:35.961853 13433 raft_consensus.cc:482] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:35.961851 13243 raft_consensus.cc:2229] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:35.962546 13243 raft_consensus.cc:2258] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:35.964429 13246 ts_tablet_manager.cc:1888] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:35.970307 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_82d6bee639334fca838d2adf031efd21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.970218 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_20dd6a8887b64221a606d7c8f804ef2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.970867 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82d6bee639334fca838d2adf031efd21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.970896 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20dd6a8887b64221a606d7c8f804ef2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.969954 13092 tablet_service.cc:1467] Processing CreateTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 (DEFAULT_TABLE table=default.Table_d5bb6a8c548e46ecb59035f49816ad52 [id=70785b5175b34eb68dc3fcba2255645b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.971834 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f3e7df09f347bfa5cad4de34dbda13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.972155 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bca67f775cc9457cb0bbcd9b659322e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.972654 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bca67f775cc9457cb0bbcd9b659322e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.973817 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8de45342e9fc4eaa8efcb1c4d73e818a" } new_table_name: "default.Table_f89d91417e7c4a6e9ede8f6c8548ee81" modify_external_catalogs: true
I20240506 19:56:35.970331 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 2417a535329341d8bb9f8218770f890d (DEFAULT_TABLE table=default.Table_c57d5247a9fc4628ac83a4fd916f2b53 [id=0db49a8d38af446f8729cfa002006438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.977716 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2417a535329341d8bb9f8218770f890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.978441 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 (DEFAULT_TABLE table=default.Table_3a6afcad38024709a1f104fa5bbadcff [id=2249bb6d1f824bacbd5b5b1c86e58c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.970665 13100 tablet_service.cc:1467] Processing CreateTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 (DEFAULT_TABLE table=default.Table_b4ce4fd613b049df9285a95fe23671c1 [id=89681fb9b58c4f429a4f94873f3827ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.979238 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5a46b91cb742a994496648fbf87aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.979627 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdab4110236e4f5c8690f6d963ff27f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.980003 13098 tablet_service.cc:1467] Processing CreateTablet for tablet cd57563cc6c140c184d1725a2da6a475 (DEFAULT_TABLE table=default.Table_b53ae7ee595f4d70813b67094f238eca [id=6a6084dba9e54d238890a1c9be17fdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.980372 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd57563cc6c140c184d1725a2da6a475. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.986261 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_052649a7995c41f2a0dde09686115a96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:35.986680 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9608d1af549c42e28d02df6bd141adf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:35.986902 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_052649a7995c41f2a0dde09686115a96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:35.987219 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9608d1af549c42e28d02df6bd141adf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:35.989920 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 40ece0ad4df247eb9d380ed1c755ac99 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.990208 13246 log.cc:1196] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/44de7153689b4dbd80dae87d3ee48a83
I20240506 19:56:35.990872 13094 tablet_service.cc:1467] Processing CreateTablet for tablet f75467ef60504763a67fc10e8639716c (DEFAULT_TABLE table=default.Table_b058641e17b2474785e2066bd9781690 [id=d64aa72e3aea448196fa09b04a0dfe7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.991561 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed (DEFAULT_TABLE table=default.Table_915828e8f55f4e1687a45a7fe7fa4f34 [id=e3a4e004e5c64784afdcc92d35a67b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.991654 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75467ef60504763a67fc10e8639716c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.993158 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3d13322b4a4a2db3b6d425679f68ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.997227 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0e2005ef58fd4fcea16abe771ee385f8
I20240506 19:56:35.950943 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34e128329204418b7d6fb55762fb590" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:35.998175 12848 raft_consensus.cc:2454] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:35.952795 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe with delete_type TABLET_DATA_DELETED (replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.959726 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a223c33ba56c4ddbba46fedbce660ae2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:35.968286 12825 tablet_service.cc:1467] Processing CreateTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 (DEFAULT_TABLE table=default.Table_d5bb6a8c548e46ecb59035f49816ad52 [id=70785b5175b34eb68dc3fcba2255645b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.999294 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f3e7df09f347bfa5cad4de34dbda13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.968786 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 2417a535329341d8bb9f8218770f890d (DEFAULT_TABLE table=default.Table_c57d5247a9fc4628ac83a4fd916f2b53 [id=0db49a8d38af446f8729cfa002006438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.001646 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2417a535329341d8bb9f8218770f890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.003886 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 07755e37a20646a3b1a7b900d54a077c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:35.969107 12822 tablet_service.cc:1467] Processing CreateTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 (DEFAULT_TABLE table=default.Table_b4ce4fd613b049df9285a95fe23671c1 [id=89681fb9b58c4f429a4f94873f3827ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.006033 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdab4110236e4f5c8690f6d963ff27f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.006472 13433 raft_consensus.cc:3037] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.006875 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:35.976704 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 (DEFAULT_TABLE table=default.Table_3a6afcad38024709a1f104fa5bbadcff [id=2249bb6d1f824bacbd5b5b1c86e58c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.007634 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9d0279aabb15419b8d8ff8dbc36f087f" } new_table_name: "default.Table_fe77d7d403a041b4b9f6a1c97177f461" modify_external_catalogs: true
I20240506 19:56:36.007882 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5a46b91cb742a994496648fbf87aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.977587 12817 tablet_service.cc:1467] Processing CreateTablet for tablet f75467ef60504763a67fc10e8639716c (DEFAULT_TABLE table=default.Table_b058641e17b2474785e2066bd9781690 [id=d64aa72e3aea448196fa09b04a0dfe7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.008669 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75467ef60504763a67fc10e8639716c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:35.979437 12815 tablet_service.cc:1467] Processing CreateTablet for tablet cd57563cc6c140c184d1725a2da6a475 (DEFAULT_TABLE table=default.Table_b53ae7ee595f4d70813b67094f238eca [id=6a6084dba9e54d238890a1c9be17fdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.010267 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.010422 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd57563cc6c140c184d1725a2da6a475. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.010757 13246 ts_tablet_manager.cc:1909] T 44de7153689b4dbd80dae87d3ee48a83 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.010991 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ffcc8db8756e4baf9a6595678ba1b3eb" } new_table_name: "default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d" modify_external_catalogs: true
W20240506 19:56:36.011655 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ba2365d00044c32a06f9017a207adf7 consensus state has no leader
I20240506 19:56:36.012941 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_74ffa3bb4ed14490a8fcf4eb9e29a709"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.013500 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ffa3bb4ed14490a8fcf4eb9e29a709 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.013661 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 44de7153689b4dbd80dae87d3ee48a83 (table default.Table_76747eb355e74563ad356ac0b78eeb8e [id=4ef402284c534d928551d10f8857c666]) successfully deleted
I20240506 19:56:36.014457 13246 tablet_replica.cc:330] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.014652 13246 raft_consensus.cc:2229] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.017803 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_712005444bfb41819b6c7f2e3a33f723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.018342 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_712005444bfb41819b6c7f2e3a33f723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.020753 13246 raft_consensus.cc:2258] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.021040 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_38260663814f40939e221b4885ed23a3" } new_table_name: "default.Table_e24bec7e49144632ab0da2f0ae398f9a" modify_external_catalogs: true
I20240506 19:56:36.021364 13246 ts_tablet_manager.cc:1875] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.022773 13245 tablet_bootstrap.cc:492] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.022995 13245 ts_tablet_manager.cc:1367] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.009s
I20240506 19:56:36.024047 13245 raft_consensus.cc:348] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.024363 13245 raft_consensus.cc:374] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.024513 13245 raft_consensus.cc:729] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.024853 13245 consensus_queue.cc:260] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:36.025992 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03e896b3ba7c4864a26cdfce76fadcfb consensus state has no leader
I20240506 19:56:36.027848 13245 ts_tablet_manager.cc:1398] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:36.028162 13245 tablet_bootstrap.cc:492] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:36.028594 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41faccd54f3e49ddb48ea5234f6cdab5 consensus state has no leader
I20240506 19:56:36.030943 13245 tablet_bootstrap.cc:654] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.035598 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_62b69a40ec6a432186ff79fc5693659e [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 5 got 0
W20240506 19:56:36.036315 12506 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet db13b58d17c94e0f82052f823915d9af (table default.Table_e882abe9665a4511bd29de22b6256710 [id=21f83b4858344bf9959aee15e02a7bf7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet db13b58d17c94e0f82052f823915d9af consensus state has no leader
I20240506 19:56:36.036814 13246 ts_tablet_manager.cc:1888] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.036950 13246 log.cc:1196] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/79486583025c42bfb1321f9cb6dd96f1
I20240506 19:56:36.037432 13246 ts_tablet_manager.cc:1909] T 79486583025c42bfb1321f9cb6dd96f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.038887 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8ae600b2261b45e8ba254d1767420f77" } new_table_name: "default.Table_f387dcc396c544b0a400bb57e5e50d05" modify_external_catalogs: true
I20240506 19:56:36.041168 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_62b69a40ec6a432186ff79fc5693659e [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 5 got 0
I20240506 19:56:36.042759 13245 tablet_bootstrap.cc:492] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.042937 13245 ts_tablet_manager.cc:1367] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240506 19:56:36.043715 13245 raft_consensus.cc:348] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.043956 13245 raft_consensus.cc:374] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.044034 13245 raft_consensus.cc:729] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.044322 13245 consensus_queue.cc:260] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.044418 12817 tablet_service.cc:1467] Processing CreateTablet for tablet b2a5a5e193f245de96cc82986e292f05 (DEFAULT_TABLE table=default.Table_2f78de17b14b4a3c85876eb23bee958a [id=f83571d593d64e58876888bfb31ccb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.047991 13245 ts_tablet_manager.cc:1398] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240506 19:56:36.048043 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:36.050347 13245 tablet_bootstrap.cc:492] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.053560 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:36.049258 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet e17197adc6ca4593ac3fc494548a6e67 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:36.058311 13245 tablet_bootstrap.cc:654] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:35.988667 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed (DEFAULT_TABLE table=default.Table_915828e8f55f4e1687a45a7fe7fa4f34 [id=e3a4e004e5c64784afdcc92d35a67b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:35.983567 13244 tablet_bootstrap.cc:654] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.058904 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a5a5e193f245de96cc82986e292f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.059523 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3d13322b4a4a2db3b6d425679f68ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.059610 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 79486583025c42bfb1321f9cb6dd96f1 (table default.Table_5f69164a5a01441fa6dc72d34b878eb2 [id=bc2b2ac5e9704dd48c4cd3620e7074e3]) successfully deleted
I20240506 19:56:36.055053 13242 log.cc:1196] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/79486583025c42bfb1321f9cb6dd96f1
I20240506 19:56:36.059967 13246 tablet_replica.cc:330] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.060168 13242 ts_tablet_manager.cc:1909] T 79486583025c42bfb1321f9cb6dd96f1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.060194 13246 raft_consensus.cc:2229] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.060361 13246 raft_consensus.cc:2258] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.060900 13246 ts_tablet_manager.cc:1875] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.061197 13242 tablet_replica.cc:330] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:36.062225 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 79486583025c42bfb1321f9cb6dd96f1 (table default.Table_5f69164a5a01441fa6dc72d34b878eb2 [id=bc2b2ac5e9704dd48c4cd3620e7074e3]) successfully deleted
W20240506 19:56:36.063913 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4be929f0e72e4d22a9af2fe6b3072bbe consensus state has no leader
I20240506 19:56:36.064644 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:36.065150 12514 catalog_manager.cc:6554] ReplaceTablet: tablet 07755e37a20646a3b1a7b900d54a077c deleted and replaced by tablet 034d6309de0740d0a5806b3847e334cb
I20240506 19:56:36.061434 13242 raft_consensus.cc:2229] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.065481 13242 raft_consensus.cc:2258] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.065712 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 07755e37a20646a3b1a7b900d54a077c with delete_type TABLET_DATA_DELETED (replaced by tablet 034d6309de0740d0a5806b3847e334cb) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.067065 13242 ts_tablet_manager.cc:1875] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:36.069646 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fce272e5b5f6483c9d031dff1104401b consensus state has no leader
I20240506 19:56:36.071125 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e17197adc6ca4593ac3fc494548a6e67
I20240506 19:56:36.071625 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 07755e37a20646a3b1a7b900d54a077c with delete_type TABLET_DATA_DELETED (replaced by tablet 034d6309de0740d0a5806b3847e334cb) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.072738 12522 catalog_manager.cc:6554] ReplaceTablet: tablet e17197adc6ca4593ac3fc494548a6e67 deleted and replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51
I20240506 19:56:36.072978 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet e17197adc6ca4593ac3fc494548a6e67 with delete_type TABLET_DATA_DELETED (replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.073241 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet e17197adc6ca4593ac3fc494548a6e67 with delete_type TABLET_DATA_DELETED (replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.074360 13101 tablet_service.cc:1467] Processing CreateTablet for tablet b2a5a5e193f245de96cc82986e292f05 (DEFAULT_TABLE table=default.Table_2f78de17b14b4a3c85876eb23bee958a [id=f83571d593d64e58876888bfb31ccb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.074945 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a5a5e193f245de96cc82986e292f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.075135 13353 raft_consensus.cc:482] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.075335 12921 leader_election.cc:302] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:36.075582 12921 raft_consensus.cc:2670] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:36.075537 12822 tablet_service.cc:1467] Processing CreateTablet for tablet fce272e5b5f6483c9d031dff1104401b (DEFAULT_TABLE table=default.Table_f387dcc396c544b0a400bb57e5e50d05 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:36.075814 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 76864.4us
I20240506 19:56:36.075944 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce272e5b5f6483c9d031dff1104401b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:36.076088 12921 leader_election.cc:341] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.076373 13436 raft_consensus.cc:482] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.076436 13246 ts_tablet_manager.cc:1888] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.076599 13246 log.cc:1196] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/8596757ef75e433fb801f7679db5fefb
I20240506 19:56:36.076994 13246 ts_tablet_manager.cc:1909] T 8596757ef75e433fb801f7679db5fefb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.077005 13434 raft_consensus.cc:482] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.077126 13434 raft_consensus.cc:504] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.078030 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 8596757ef75e433fb801f7679db5fefb (table default.Table_73b96b599455444fa8fead40db334a12 [id=a76c2b5f25a24c55a8379608a9857eda]) successfully deleted
I20240506 19:56:36.079461 13434 leader_election.cc:288] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.080037 13353 raft_consensus.cc:504] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:36.080740 12922 leader_election.cc:341] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8596757ef75e433fb801f7679db5fefb
I20240506 19:56:36.081029 12962 tablet_service.cc:1467] Processing CreateTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 (DEFAULT_TABLE table=default.Table_d5bb6a8c548e46ecb59035f49816ad52 [id=70785b5175b34eb68dc3fcba2255645b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.081683 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe1c6ed89d643ecb4e97402d78167b6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.083173 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f3e7df09f347bfa5cad4de34dbda13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.083802 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 2417a535329341d8bb9f8218770f890d (DEFAULT_TABLE table=default.Table_c57d5247a9fc4628ac83a4fd916f2b53 [id=0db49a8d38af446f8729cfa002006438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.084160 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2417a535329341d8bb9f8218770f890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.084519 13246 tablet_replica.cc:330] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.084607 13101 tablet_service.cc:1467] Processing CreateTablet for tablet fce272e5b5f6483c9d031dff1104401b (DEFAULT_TABLE table=default.Table_f387dcc396c544b0a400bb57e5e50d05 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.084781 13246 raft_consensus.cc:2229] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:36.086084 12922 leader_election.cc:341] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.085263 13353 leader_election.cc:288] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.086438 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad527c3f2a4b53affa928e149db1df (DEFAULT_TABLE table=default.Table_0e976f7b85cd40e49f346f31b8a5d1ee [id=cdb5ce64abc74bb3bf60d7aa727c515f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.087095 13249 ts_tablet_manager.cc:1398] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.056s user 0.001s sys 0.002s
I20240506 19:56:36.087121 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8596757ef75e433fb801f7679db5fefb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.087146 12961 tablet_service.cc:1467] Processing CreateTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 (DEFAULT_TABLE table=default.Table_b4ce4fd613b049df9285a95fe23671c1 [id=89681fb9b58c4f429a4f94873f3827ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.087531 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe1c6ed89d643ecb4e97402d78167b6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.088626 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad527c3f2a4b53affa928e149db1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.089885 13249 tablet_bootstrap.cc:492] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.085018 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce272e5b5f6483c9d031dff1104401b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.090548 13243 ts_tablet_manager.cc:1875] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.076500 13436 raft_consensus.cc:504] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.091001 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdab4110236e4f5c8690f6d963ff27f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.094590 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89eb4e2701894f32a85d2cd89da58ec2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.095521 12957 tablet_service.cc:1467] Processing CreateTablet for tablet cd57563cc6c140c184d1725a2da6a475 (DEFAULT_TABLE table=default.Table_b53ae7ee595f4d70813b67094f238eca [id=6a6084dba9e54d238890a1c9be17fdec]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:36.096211 12921 leader_election.cc:341] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.096607 12967 tablet_service.cc:1467] Processing CreateTablet for tablet b2a5a5e193f245de96cc82986e292f05 (DEFAULT_TABLE table=default.Table_2f78de17b14b4a3c85876eb23bee958a [id=f83571d593d64e58876888bfb31ccb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.097101 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 (DEFAULT_TABLE table=default.Table_3a6afcad38024709a1f104fa5bbadcff [id=2249bb6d1f824bacbd5b5b1c86e58c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.097690 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.087070 13246 raft_consensus.cc:2258] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.098038 12921 leader_election.cc:302] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.098249 13433 raft_consensus.cc:504] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.098901 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed (DEFAULT_TABLE table=default.Table_915828e8f55f4e1687a45a7fe7fa4f34 [id=e3a4e004e5c64784afdcc92d35a67b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.099157 13433 leader_election.cc:288] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.099342 13436 leader_election.cc:288] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.100426 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad527c3f2a4b53affa928e149db1df (DEFAULT_TABLE table=default.Table_0e976f7b85cd40e49f346f31b8a5d1ee [id=cdb5ce64abc74bb3bf60d7aa727c515f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.100858 12960 tablet_service.cc:1467] Processing CreateTablet for tablet fce272e5b5f6483c9d031dff1104401b (DEFAULT_TABLE table=default.Table_f387dcc396c544b0a400bb57e5e50d05 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.101348 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad527c3f2a4b53affa928e149db1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.102233 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 07755e37a20646a3b1a7b900d54a077c with delete_type TABLET_DATA_DELETED (replaced by tablet 034d6309de0740d0a5806b3847e334cb) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:36.102556 12922 leader_election.cc:341] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.102996 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_685d0e00c37640ab99bfe74c85c1c0f3" } modify_external_catalogs: true
I20240506 19:56:36.103210 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_685d0e00c37640ab99bfe74c85c1c0f3" } modify_external_catalogs: true
I20240506 19:56:36.103341 13436 raft_consensus.cc:2726] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:36.103459 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a223c33ba56c4ddbba46fedbce660ae2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:36.103991 12922 leader_election.cc:341] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.105105 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet e17197adc6ca4593ac3fc494548a6e67 with delete_type TABLET_DATA_DELETED (replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.105669 13246 ts_tablet_manager.cc:1875] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.109450 12956 tablet_service.cc:1467] Processing CreateTablet for tablet f75467ef60504763a67fc10e8639716c (DEFAULT_TABLE table=default.Table_b058641e17b2474785e2066bd9781690 [id=d64aa72e3aea448196fa09b04a0dfe7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.109978 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd57563cc6c140c184d1725a2da6a475. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:36.110736 12921 leader_election.cc:341] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.110896 12921 leader_election.cc:302] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.112165 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89eb4e2701894f32a85d2cd89da58ec2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.112624 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a223c33ba56c4ddbba46fedbce660ae2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.117220 13246 ts_tablet_manager.cc:1888] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.117311 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ea25fb8e42234b82885b8fb2275d4925
I20240506 19:56:36.118084 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b4ce4fd613b049df9285a95fe23671c1" } modify_external_catalogs: true
I20240506 19:56:36.118294 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b4ce4fd613b049df9285a95fe23671c1" } modify_external_catalogs: true
I20240506 19:56:36.118526 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad527c3f2a4b53affa928e149db1df (DEFAULT_TABLE table=default.Table_0e976f7b85cd40e49f346f31b8a5d1ee [id=cdb5ce64abc74bb3bf60d7aa727c515f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.119031 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_53b38bd9c00f46f2abac3c4ca498fb0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.119575 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53b38bd9c00f46f2abac3c4ca498fb0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.119668 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3d13322b4a4a2db3b6d425679f68ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.119670 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet ea25fb8e42234b82885b8fb2275d4925 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.120015 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d5bb6a8c548e46ecb59035f49816ad52" } modify_external_catalogs: true
I20240506 19:56:36.120198 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d5bb6a8c548e46ecb59035f49816ad52" } modify_external_catalogs: true
I20240506 19:56:36.120859 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_038664bd22da4e20b40b21843516298f" } modify_external_catalogs: true
I20240506 19:56:36.121047 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_038664bd22da4e20b40b21843516298f" } modify_external_catalogs: true
I20240506 19:56:36.122454 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ba47b7bc355a4589a2b439289c6a61de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.122942 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba47b7bc355a4589a2b439289c6a61de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.123569 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (DEFAULT_TABLE table=default.Table_cef67076343c4a73b29b2d8091257dd2 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.124004 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b2a4720f6a46e4b0412a58c1e26bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.124024 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a5a5e193f245de96cc82986e292f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.124269 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce272e5b5f6483c9d031dff1104401b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.124505 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75467ef60504763a67fc10e8639716c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:36.124446 12921 leader_election.cc:341] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.124630 12921 leader_election.cc:302] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.124693 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5a46b91cb742a994496648fbf87aa4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:36.125135 12921 leader_election.cc:341] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.125385 13436 raft_consensus.cc:2726] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:36.125599 13436 raft_consensus.cc:2726] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:36.122423 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e3428b42c94e4d8ca3620c86ea4d3082"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.126284 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3428b42c94e4d8ca3620c86ea4d3082 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.125276 12921 leader_election.cc:302] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.117601 13246 log.cc:1196] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a223c33ba56c4ddbba46fedbce660ae2
I20240506 19:56:36.132223 13246 ts_tablet_manager.cc:1909] T a223c33ba56c4ddbba46fedbce660ae2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.133891 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet fdab4110236e4f5c8690f6d963ff27f7
I20240506 19:56:36.134784 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.121366 13329 raft_consensus.cc:482] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.135097 13329 raft_consensus.cc:504] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.135915 13329 leader_election.cc:288] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.136132 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.122825 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet ea25fb8e42234b82885b8fb2275d4925 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.136312 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e896b3ba7c4864a26cdfce76fadcfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.136492 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 581aae0889654903933bef3bf1026954 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:36.136538 12847 raft_consensus.cc:2454] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:36.137878 13058 leader_election.cc:302] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:36.138597 13329 raft_consensus.cc:2781] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.138700 13329 raft_consensus.cc:482] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.138795 13329 raft_consensus.cc:3037] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.139889 13433 raft_consensus.cc:2726] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:36.140641 13436 raft_consensus.cc:482] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.140810 13436 raft_consensus.cc:504] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.140975 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet ea25fb8e42234b82885b8fb2275d4925 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.142161 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30cb3cb7cbe14cc9bf1bfd07227f538e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.142607 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad527c3f2a4b53affa928e149db1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.145565 13242 ts_tablet_manager.cc:1888] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.145776 13242 log.cc:1196] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/f017e635010040008be72b3813b5a4ed
I20240506 19:56:36.146205 13242 ts_tablet_manager.cc:1909] T f017e635010040008be72b3813b5a4ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.146842 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet f017e635010040008be72b3813b5a4ed (table default.Table_bb529dd8d37f40449f27b0e07cbc93dc [id=4b9a2f4ea4b1460f89e8981e952e0d2d]) successfully deleted
I20240506 19:56:36.149554 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a223c33ba56c4ddbba46fedbce660ae2 (table default.Table_218a52f567cf4031bee1b7834387ac84 [id=779ea3288b6d40ceb9636fdde3697472]) successfully deleted
W20240506 19:56:36.150456 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 20c1590e36b4471d8e72f4b07410ee0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20c1590e36b4471d8e72f4b07410ee0f already in progress: creating tablet
I20240506 19:56:36.150826 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f0a8b12f36e74c95ad6eaf30e432f3e9
W20240506 19:56:36.151208 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 40ece0ad4df247eb9d380ed1c755ac99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40ece0ad4df247eb9d380ed1c755ac99 already in progress: creating tablet
I20240506 19:56:36.151412 13329 raft_consensus.cc:504] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.151893 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.152010 13246 tablet_replica.cc:330] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.152251 13329 leader_election.cc:288] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.152312 13246 raft_consensus.cc:2229] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.152536 13246 raft_consensus.cc:2258] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.152554 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d8f3e7df09f347bfa5cad4de34dbda13
I20240506 19:56:36.152626 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e896b3ba7c4864a26cdfce76fadcfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.152902 12847 raft_consensus.cc:3037] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.153352 13246 ts_tablet_manager.cc:1875] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.159436 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 581aae0889654903933bef3bf1026954
I20240506 19:56:36.159862 12522 catalog_manager.cc:6554] ReplaceTablet: tablet 581aae0889654903933bef3bf1026954 deleted and replaced by tablet 63f56d2180b1472a8168fb3a63589065
I20240506 19:56:36.160596 13246 ts_tablet_manager.cc:1888] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.160880 13246 log.cc:1196] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/8842523a97954e3f81ffd87530201d93
I20240506 19:56:36.161264 13246 ts_tablet_manager.cc:1909] T 8842523a97954e3f81ffd87530201d93 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.161659 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (DEFAULT_TABLE table=default.Table_cef67076343c4a73b29b2d8091257dd2 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.162107 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b2a4720f6a46e4b0412a58c1e26bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.162950 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 581aae0889654903933bef3bf1026954 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f56d2180b1472a8168fb3a63589065) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.163079 13242 tablet_replica.cc:330] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:36.163298 13242 raft_consensus.cc:2229] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.163470 13242 raft_consensus.cc:2258] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.163947 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.164094 13242 ts_tablet_manager.cc:1875] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.164448 12847 raft_consensus.cc:2454] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:36.164991 13058 leader_election.cc:302] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:36.166107 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 8842523a97954e3f81ffd87530201d93 (table default.Table_0594be752cc441b1af65142e6d925488 [id=b1277bb384294e7ea3cd5d0a52b5c813]) successfully deleted
I20240506 19:56:36.166154 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 104337875517445f862d82ece3a84889 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:36.166419 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet e34e128329204418b7d6fb55762fb590 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e34e128329204418b7d6fb55762fb590 already in progress: creating tablet
W20240506 19:56:36.166633 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 88dba275893f4696a5318ec21f0b2477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88dba275893f4696a5318ec21f0b2477 already in progress: creating tablet
I20240506 19:56:36.166754 13436 leader_election.cc:288] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.167531 13329 raft_consensus.cc:2781] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.167853 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.167879 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 581aae0889654903933bef3bf1026954 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f56d2180b1472a8168fb3a63589065) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.168022 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:36.168706 12922 leader_election.cc:341] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 30cb3cb7cbe14cc9bf1bfd07227f538e
I20240506 19:56:36.170751 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:36.171072 12921 leader_election.cc:341] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.171525 12921 leader_election.cc:302] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.172155 13242 ts_tablet_manager.cc:1888] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.172220 13244 tablet_bootstrap.cc:492] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.172310 13242 log.cc:1196] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/44de7153689b4dbd80dae87d3ee48a83
I20240506 19:56:36.172380 13244 ts_tablet_manager.cc:1367] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.229s user 0.005s sys 0.009s
I20240506 19:56:36.171926 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e896b3ba7c4864a26cdfce76fadcfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:36.172727 13242 ts_tablet_manager.cc:1909] T 44de7153689b4dbd80dae87d3ee48a83 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.172767 12988 raft_consensus.cc:2454] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:36.172729 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e896b3ba7c4864a26cdfce76fadcfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:36.173239 13244 raft_consensus.cc:348] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.173820 13244 raft_consensus.cc:374] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.173913 13244 raft_consensus.cc:729] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.173977 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 44de7153689b4dbd80dae87d3ee48a83 (table default.Table_76747eb355e74563ad356ac0b78eeb8e [id=4ef402284c534d928551d10f8857c666]) successfully deleted
I20240506 19:56:36.174620 13244 consensus_queue.cc:260] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:36.175055 12507 catalog_manager.cc:4797] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b5adab0af6274ba0ad54402c5b1ff757 because the tablet was not found. No further retry: Not found: Tablet not found: b5adab0af6274ba0ad54402c5b1ff757
I20240506 19:56:36.175256 13249 tablet_bootstrap.cc:654] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:36.175705 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 20c1590e36b4471d8e72f4b07410ee0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20c1590e36b4471d8e72f4b07410ee0f already in progress: creating tablet
W20240506 19:56:36.176240 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 40ece0ad4df247eb9d380ed1c755ac99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40ece0ad4df247eb9d380ed1c755ac99 already in progress: creating tablet
I20240506 19:56:36.176544 12987 raft_consensus.cc:3037] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.176893 13436 raft_consensus.cc:2726] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:36.176918 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 88dba275893f4696a5318ec21f0b2477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88dba275893f4696a5318ec21f0b2477 already in progress: creating tablet
I20240506 19:56:36.177201 13242 tablet_replica.cc:330] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:36.177438 13242 raft_consensus.cc:2229] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.177775 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (DEFAULT_TABLE table=default.Table_cef67076343c4a73b29b2d8091257dd2 [id=1ef16dfbd249466fa6b0a7df43515b7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:36.178988 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2751074a54ed4bcea6979e6ad11c2efe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2751074a54ed4bcea6979e6ad11c2efe already in progress: creating tablet
I20240506 19:56:36.179829 13246 tablet_replica.cc:330] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.180110 13246 raft_consensus.cc:2229] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.180263 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b2a4720f6a46e4b0412a58c1e26bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.180759 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.180977 13246 raft_consensus.cc:2258] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.182137 13245 tablet_bootstrap.cc:492] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.182281 13245 ts_tablet_manager.cc:1367] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.132s user 0.009s sys 0.009s
I20240506 19:56:36.182912 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.183090 13245 raft_consensus.cc:348] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.183306 13245 raft_consensus.cc:374] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.183403 13245 raft_consensus.cc:729] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.183681 13245 consensus_queue.cc:260] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.184837 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 581aae0889654903933bef3bf1026954 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f56d2180b1472a8168fb3a63589065) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.188118 13242 raft_consensus.cc:2258] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.188690 13244 ts_tablet_manager.cc:1398] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240506 19:56:36.189096 13244 tablet_bootstrap.cc:492] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.193137 13242 ts_tablet_manager.cc:1875] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.193503 13246 ts_tablet_manager.cc:1875] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.198649 12987 raft_consensus.cc:2454] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:36.204355 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.204586 13243 ts_tablet_manager.cc:1888] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.204119 13329 raft_consensus.cc:686] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:36.205185 13243 log.cc:1196] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/e34e128329204418b7d6fb55762fb590
I20240506 19:56:36.204984 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e5d18039b7f04edaa0c985be5260be35" } modify_external_catalogs: true
I20240506 19:56:36.205653 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e5d18039b7f04edaa0c985be5260be35" } modify_external_catalogs: true
I20240506 19:56:36.205416 13329 consensus_queue.cc:237] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.206734 13244 tablet_bootstrap.cc:654] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.208479 13245 ts_tablet_manager.cc:1398] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
I20240506 19:56:36.210552 13243 ts_tablet_manager.cc:1909] T e34e128329204418b7d6fb55762fb590 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.211266 13245 tablet_bootstrap.cc:492] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.210573 13242 ts_tablet_manager.cc:1888] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.211609 13243 tablet_replica.cc:330] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:36.211802 13242 log.cc:1196] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/e34e128329204418b7d6fb55762fb590
I20240506 19:56:36.211793 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet e34e128329204418b7d6fb55762fb590 (table default.Table_f44dcd3362d64342ad02f7d17b3f24d0 [id=a147e46c173c402db02793b1e11de5b5]) successfully deleted
I20240506 19:56:36.213600 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c3e795b5a9724754a33f79ebe9ac2c1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.214326 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3e795b5a9724754a33f79ebe9ac2c1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.214408 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_788bbfde3ac54d13b047fa3c065680d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.214946 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_788bbfde3ac54d13b047fa3c065680d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.215425 13245 tablet_bootstrap.cc:654] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.215639 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a410467878c2481f86118d12ccd06f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.216112 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a410467878c2481f86118d12ccd06f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.219334 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_56eb06f51079472894642f592bbe43a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.219838 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56eb06f51079472894642f592bbe43a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.220679 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6974ae6398ae4559a50fb236133424b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.221199 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6974ae6398ae4559a50fb236133424b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.222121 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c7ba3500fa6c464996810bc6a6a1af48" } new_table_name: "default.Table_01b7228cdeff4166bcd9d1480dbcc479" modify_external_catalogs: true
I20240506 19:56:36.224159 13246 ts_tablet_manager.cc:1888] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.224292 13246 log.cc:1196] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d843fe82af1148f4a9045ce8b09ad11d
I20240506 19:56:36.224637 13246 ts_tablet_manager.cc:1909] T d843fe82af1148f4a9045ce8b09ad11d P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.230775 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f7e74cbd94724e53803a5429d2079986" } new_table_name: "default.Table_0778153fb8284502a1f3e86332825c48" modify_external_catalogs: true
I20240506 19:56:36.232801 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_92531c4a95d14b50b186416c9cfd94b8" } new_table_name: "default.Table_ca61f3a9aefa4fba8fea32d717de3fb1" modify_external_catalogs: true
I20240506 19:56:36.233873 13242 ts_tablet_manager.cc:1909] T e34e128329204418b7d6fb55762fb590 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.236143 13244 tablet_bootstrap.cc:492] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.236356 13244 ts_tablet_manager.cc:1367] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.001s
I20240506 19:56:36.236143 13245 tablet_bootstrap.cc:492] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.237069 13245 ts_tablet_manager.cc:1367] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20240506 19:56:36.237522 13242 tablet_replica.cc:330] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:36.237488 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet e34e128329204418b7d6fb55762fb590 (table default.Table_f44dcd3362d64342ad02f7d17b3f24d0 [id=a147e46c173c402db02793b1e11de5b5]) successfully deleted
I20240506 19:56:36.237780 13242 raft_consensus.cc:2229] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.237973 13242 raft_consensus.cc:2258] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.239059 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d843fe82af1148f4a9045ce8b09ad11d (table default.Table_bbc623ea167c48929e77cee2b3d07aaa [id=f6f5f08248a142d6929736e7872ae027]) successfully deleted
I20240506 19:56:36.239132 13244 raft_consensus.cc:348] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.239364 13244 raft_consensus.cc:374] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.239456 13244 raft_consensus.cc:729] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.239732 13244 consensus_queue.cc:260] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:36.240672 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0e2005ef58fd4fcea16abe771ee385f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e2005ef58fd4fcea16abe771ee385f8 already in progress: creating tablet
I20240506 19:56:36.243194 13245 raft_consensus.cc:348] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.244519 13245 raft_consensus.cc:374] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.244822 13245 raft_consensus.cc:729] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.245505 13245 consensus_queue.cc:260] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:36.246232 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 07755e37a20646a3b1a7b900d54a077c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07755e37a20646a3b1a7b900d54a077c already in progress: creating tablet
W20240506 19:56:36.246555 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet e17197adc6ca4593ac3fc494548a6e67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e17197adc6ca4593ac3fc494548a6e67 already in progress: creating tablet
W20240506 19:56:36.246771 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet fdab4110236e4f5c8690f6d963ff27f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdab4110236e4f5c8690f6d963ff27f7 already in progress: creating tablet
W20240506 19:56:36.247030 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ea25fb8e42234b82885b8fb2275d4925 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea25fb8e42234b82885b8fb2275d4925 already in progress: creating tablet
I20240506 19:56:36.247448 13245 ts_tablet_manager.cc:1398] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:36.249122 13245 tablet_bootstrap.cc:492] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.249166 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d6d662a23cfb4d02be80346ea62fbe2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:36.249538 13436 raft_consensus.cc:482] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.249706 13436 raft_consensus.cc:504] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:36.249974 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6d662a23cfb4d02be80346ea62fbe2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.250432 13436 leader_election.cc:288] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.251194 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b7e42061cd487b8517447eb59ea8a1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.251685 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b7e42061cd487b8517447eb59ea8a1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.253335 13245 tablet_bootstrap.cc:654] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:36.253363 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0a8b12f36e74c95ad6eaf30e432f3e9 already in progress: creating tablet
W20240506 19:56:36.253636 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d8f3e7df09f347bfa5cad4de34dbda13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8f3e7df09f347bfa5cad4de34dbda13 already in progress: creating tablet
W20240506 19:56:36.253859 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 581aae0889654903933bef3bf1026954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 581aae0889654903933bef3bf1026954 already in progress: creating tablet
W20240506 19:56:36.254241 12921 leader_election.cc:341] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.256798 13243 raft_consensus.cc:2229] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.257061 13243 raft_consensus.cc:2258] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:36.258464 12922 leader_election.cc:341] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.259711 12922 leader_election.cc:302] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.260242 13436 raft_consensus.cc:2726] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:36.264339 13244 ts_tablet_manager.cc:1398] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240506 19:56:36.264767 13244 tablet_bootstrap.cc:492] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.268425 13244 tablet_bootstrap.cc:654] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.270433 13245 tablet_bootstrap.cc:492] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.270604 13245 ts_tablet_manager.cc:1367] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20240506 19:56:36.271386 13245 raft_consensus.cc:348] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.273914 13245 raft_consensus.cc:374] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.274048 13245 raft_consensus.cc:729] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.274300 13245 consensus_queue.cc:260] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.276297 13245 ts_tablet_manager.cc:1398] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:36.276628 13245 tablet_bootstrap.cc:492] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.277956 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet 4be929f0e72e4d22a9af2fe6b3072bbe from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:36.278586 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca394b7f53ce445e8778dd25d0cf25c7 consensus state has no leader
I20240506 19:56:36.279074 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_162d0c1d88d04a11b0015afaea52e19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.279865 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_162d0c1d88d04a11b0015afaea52e19f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.283190 13245 tablet_bootstrap.cc:654] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.286131 13436 raft_consensus.cc:482] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.286350 13436 raft_consensus.cc:504] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.287189 13436 leader_election.cc:288] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.287973 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156c6f582b747de973b357b6d071a46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:36.288708 12921 leader_election.cc:341] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.289731 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156c6f582b747de973b357b6d071a46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.290038 13124 raft_consensus.cc:2454] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:36.290709 12922 leader_election.cc:302] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.291326 13436 raft_consensus.cc:2781] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.291471 13436 raft_consensus.cc:482] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.291565 13436 raft_consensus.cc:3037] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.293643 12513 catalog_manager.cc:5484] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:36.294353 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 03e896b3ba7c4864a26cdfce76fadcfb (table default.Table_fe77d7d403a041b4b9f6a1c97177f461 [id=15e39c71182c4a089e9c950679153305]). Expected version 1 got 0
I20240506 19:56:36.295584 13436 raft_consensus.cc:504] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.301098 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 378759ca310f453fba45d48879e2dd1a (DEFAULT_TABLE table=default.Table_bca67f775cc9457cb0bbcd9b659322e8 [id=95ada67289714d7d8529b5338c361fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.301741 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 378759ca310f453fba45d48879e2dd1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.301995 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156c6f582b747de973b357b6d071a46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:36.302922 12921 leader_election.cc:341] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.303678 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 (DEFAULT_TABLE table=default.Table_052649a7995c41f2a0dde09686115a96 [id=5e3d0dbf38d5451188d26847793c20a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.304165 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.305696 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 (DEFAULT_TABLE table=default.Table_052649a7995c41f2a0dde09686115a96 [id=5e3d0dbf38d5451188d26847793c20a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.306245 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.301091 12827 tablet_service.cc:1467] Processing CreateTablet for tablet ac9ecd2e034e49e49772db7435da3c81 (DEFAULT_TABLE table=default.Table_9608d1af549c42e28d02df6bd141adf8 [id=feed002cdb9d4296a14ca76514e18ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.309731 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 378759ca310f453fba45d48879e2dd1a (DEFAULT_TABLE table=default.Table_bca67f775cc9457cb0bbcd9b659322e8 [id=95ada67289714d7d8529b5338c361fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.310004 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9ecd2e034e49e49772db7435da3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.310190 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 378759ca310f453fba45d48879e2dd1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.311439 13243 ts_tablet_manager.cc:1875] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.315644 13101 tablet_service.cc:1467] Processing CreateTablet for tablet b35231d3f1bb4348aa7af3c71b809cd9 (DEFAULT_TABLE table=default.Table_74ffa3bb4ed14490a8fcf4eb9e29a709 [id=dbcaca64a73545e78c668161948a1b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.316212 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35231d3f1bb4348aa7af3c71b809cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.319445 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62b69a40ec6a432186ff79fc5693659e" } new_table_name: "default.Table_92eb5ef0ea1c408c98a0a14387f76b6e" modify_external_catalogs: true
I20240506 19:56:36.322876 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 378759ca310f453fba45d48879e2dd1a (DEFAULT_TABLE table=default.Table_bca67f775cc9457cb0bbcd9b659322e8 [id=95ada67289714d7d8529b5338c361fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.323436 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 378759ca310f453fba45d48879e2dd1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.323921 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.325105 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_20dd6a8887b64221a606d7c8f804ef2b" } new_table_name: "default.Table_d004ba3983954d4aada9c7d0e1a0207d" modify_external_catalogs: true
I20240506 19:56:36.326304 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 (DEFAULT_TABLE table=default.Table_052649a7995c41f2a0dde09686115a96 [id=5e3d0dbf38d5451188d26847793c20a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.326820 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.333971 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_82d6bee639334fca838d2adf031efd21" } modify_external_catalogs: true
I20240506 19:56:36.334308 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_82d6bee639334fca838d2adf031efd21" } modify_external_catalogs: true
I20240506 19:56:36.342178 12965 tablet_service.cc:1467] Processing CreateTablet for tablet b35231d3f1bb4348aa7af3c71b809cd9 (DEFAULT_TABLE table=default.Table_74ffa3bb4ed14490a8fcf4eb9e29a709 [id=dbcaca64a73545e78c668161948a1b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.342978 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35231d3f1bb4348aa7af3c71b809cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.349325 13329 raft_consensus.cc:482] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.349512 13329 raft_consensus.cc:504] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.350342 13329 leader_election.cc:288] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.351466 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e714ed6b64f2d84770e7182e4103a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.352578 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (DEFAULT_TABLE table=default.Table_20dd6a8887b64221a606d7c8f804ef2b [id=2c018b5df19c4415bf0aa2e45b1f7b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.353103 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.353083 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e714ed6b64f2d84770e7182e4103a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:36.353348 12987 raft_consensus.cc:2454] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:36.353335 13058 leader_election.cc:341] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.354327 12962 tablet_service.cc:1467] Processing CreateTablet for tablet ac9ecd2e034e49e49772db7435da3c81 (DEFAULT_TABLE table=default.Table_9608d1af549c42e28d02df6bd141adf8 [id=feed002cdb9d4296a14ca76514e18ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.354848 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9ecd2e034e49e49772db7435da3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.356931 13057 leader_election.cc:302] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.357401 13329 raft_consensus.cc:2781] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.357571 13329 raft_consensus.cc:482] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.357666 13329 raft_consensus.cc:3037] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.360442 13329 raft_consensus.cc:504] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.361186 13329 leader_election.cc:288] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.361824 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e714ed6b64f2d84770e7182e4103a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.362737 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e714ed6b64f2d84770e7182e4103a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
W20240506 19:56:36.362840 13058 leader_election.cc:341] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.362962 12987 raft_consensus.cc:3037] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.365885 12987 raft_consensus.cc:2454] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:36.367230 13057 leader_election.cc:302] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.367616 13329 raft_consensus.cc:2781] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.367767 13329 raft_consensus.cc:686] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:36.368105 13329 consensus_queue.cc:237] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.368450 13104 tablet_service.cc:1467] Processing CreateTablet for tablet ac9ecd2e034e49e49772db7435da3c81 (DEFAULT_TABLE table=default.Table_9608d1af549c42e28d02df6bd141adf8 [id=feed002cdb9d4296a14ca76514e18ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.369050 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9ecd2e034e49e49772db7435da3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.372788 12819 tablet_service.cc:1467] Processing CreateTablet for tablet b35231d3f1bb4348aa7af3c71b809cd9 (DEFAULT_TABLE table=default.Table_74ffa3bb4ed14490a8fcf4eb9e29a709 [id=dbcaca64a73545e78c668161948a1b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.374344 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35231d3f1bb4348aa7af3c71b809cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.378998 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (DEFAULT_TABLE table=default.Table_20dd6a8887b64221a606d7c8f804ef2b [id=2c018b5df19c4415bf0aa2e45b1f7b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.379487 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.398963 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (DEFAULT_TABLE table=default.Table_20dd6a8887b64221a606d7c8f804ef2b [id=2c018b5df19c4415bf0aa2e45b1f7b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.399638 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.400254 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156c6f582b747de973b357b6d071a46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.400549 13124 raft_consensus.cc:3037] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:36.404189 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3d57249f30a44dab3384092ec20b515 consensus state has no leader
I20240506 19:56:36.404392 13124 raft_consensus.cc:2454] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:36.405066 12922 leader_election.cc:302] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.406046 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7af138f7bd89452c9f067f9c2fc6f955" } modify_external_catalogs: true
I20240506 19:56:36.406147 13433 raft_consensus.cc:2781] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.406281 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7af138f7bd89452c9f067f9c2fc6f955" } modify_external_catalogs: true
I20240506 19:56:36.406353 13433 raft_consensus.cc:686] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:36.406701 13433 consensus_queue.cc:237] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.409670 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b50238125ad14920bf820245406918c3" } modify_external_catalogs: true
I20240506 19:56:36.409973 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b50238125ad14920bf820245406918c3" } modify_external_catalogs: true
I20240506 19:56:36.417133 13243 ts_tablet_manager.cc:1888] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.422767 13436 leader_election.cc:288] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.425635 13249 tablet_bootstrap.cc:492] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.425809 13249 ts_tablet_manager.cc:1367] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.338s user 0.017s sys 0.052s
I20240506 19:56:36.426400 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 (DEFAULT_TABLE table=default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.426899 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4095f9fc28ff48f68064ec8318ff7b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.430528 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet cc8335f9308046279a25a4a95d8e0f65 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:36.426640 13249 raft_consensus.cc:348] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.431182 13249 raft_consensus.cc:374] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.431298 13249 raft_consensus.cc:729] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.431536 13249 consensus_queue.cc:260] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.432137 13249 ts_tablet_manager.cc:1398] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240506 19:56:36.432469 13249 tablet_bootstrap.cc:492] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.434245 13249 tablet_bootstrap.cc:654] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.437613 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 17fb091762144f6591152680a6acd29e (DEFAULT_TABLE table=default.Table_82d6bee639334fca838d2adf031efd21 [id=d8e5bdbac7204b04ba040114b49d6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.438078 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17fb091762144f6591152680a6acd29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.438759 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca61f3a9aefa4fba8fea32d717de3fb1" } modify_external_catalogs: true
I20240506 19:56:36.439018 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca61f3a9aefa4fba8fea32d717de3fb1" } modify_external_catalogs: true
I20240506 19:56:36.439922 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 17fb091762144f6591152680a6acd29e (DEFAULT_TABLE table=default.Table_82d6bee639334fca838d2adf031efd21 [id=d8e5bdbac7204b04ba040114b49d6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.440363 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17fb091762144f6591152680a6acd29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.444144 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 7de7397db1e74b4488a3f9a70136bccb (DEFAULT_TABLE table=default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.444692 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de7397db1e74b4488a3f9a70136bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.445216 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 7de7397db1e74b4488a3f9a70136bccb (DEFAULT_TABLE table=default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.445623 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de7397db1e74b4488a3f9a70136bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.448195 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 (DEFAULT_TABLE table=default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.448762 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 7de7397db1e74b4488a3f9a70136bccb (DEFAULT_TABLE table=default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.449328 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de7397db1e74b4488a3f9a70136bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.451911 13252 raft_consensus.cc:482] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.452795 13252 raft_consensus.cc:504] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.453765 13252 leader_election.cc:288] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.454181 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 034d6309de0740d0a5806b3847e334cb (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.454488 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4095f9fc28ff48f68064ec8318ff7b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.455371 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89eb4e2701894f32a85d2cd89da58ec2" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.455690 13124 raft_consensus.cc:2454] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:36.457706 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034d6309de0740d0a5806b3847e334cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.457769 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89eb4e2701894f32a85d2cd89da58ec2" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:36.457993 12987 raft_consensus.cc:2454] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:36.458523 13436 raft_consensus.cc:482] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.458740 13436 raft_consensus.cc:504] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.459686 13436 leader_election.cc:288] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:36.460287 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc0ac37f67942fc8a31d8d43c839246" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.460492 13124 raft_consensus.cc:2454] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:36.460901 12922 leader_election.cc:302] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:36.461230 13436 raft_consensus.cc:2781] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.461398 13436 raft_consensus.cc:482] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.461547 13436 raft_consensus.cc:3037] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.467130 13249 tablet_bootstrap.cc:492] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.474694 13245 tablet_bootstrap.cc:492] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.474951 13245 ts_tablet_manager.cc:1367] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.198s user 0.017s sys 0.029s
I20240506 19:56:36.475966 13245 raft_consensus.cc:348] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.476284 13245 raft_consensus.cc:374] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.476382 13245 raft_consensus.cc:729] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.476806 13245 consensus_queue.cc:260] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.477722 13245 ts_tablet_manager.cc:1398] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240506 19:56:36.478335 13245 tablet_bootstrap.cc:492] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.465858 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 17fb091762144f6591152680a6acd29e (DEFAULT_TABLE table=default.Table_82d6bee639334fca838d2adf031efd21 [id=d8e5bdbac7204b04ba040114b49d6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.480170 13249 ts_tablet_manager.cc:1367] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20240506 19:56:36.480170 13243 log.cc:1196] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d843fe82af1148f4a9045ce8b09ad11d
I20240506 19:56:36.481016 13245 tablet_bootstrap.cc:654] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.482769 13249 raft_consensus.cc:348] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.484076 13249 raft_consensus.cc:374] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.484189 13249 raft_consensus.cc:729] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.485062 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 034d6309de0740d0a5806b3847e334cb (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.485634 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034d6309de0740d0a5806b3847e334cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.461052 12783 leader_election.cc:302] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:36.491174 13252 raft_consensus.cc:2781] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.491360 13252 raft_consensus.cc:482] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.491503 13252 raft_consensus.cc:3037] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.493712 13249 consensus_queue.cc:260] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.494071 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 034d6309de0740d0a5806b3847e334cb (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.494408 13249 ts_tablet_manager.cc:1398] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
I20240506 19:56:36.495401 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 (DEFAULT_TABLE table=default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:36.498553 13436 raft_consensus.cc:504] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.499572 13436 leader_election.cc:288] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:36.501250 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc0ac37f67942fc8a31d8d43c839246" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.501585 13124 raft_consensus.cc:3037] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.508375 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034d6309de0740d0a5806b3847e334cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.508121 13124 raft_consensus.cc:2454] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:36.508805 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc0ac37f67942fc8a31d8d43c839246" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.509513 13249 tablet_bootstrap.cc:492] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.509517 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc0ac37f67942fc8a31d8d43c839246" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.510464 13252 raft_consensus.cc:504] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.510548 12922 leader_election.cc:302] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:36.512017 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89eb4e2701894f32a85d2cd89da58ec2" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.512315 13124 raft_consensus.cc:3037] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:36.512774 12921 leader_election.cc:341] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.513499 13436 raft_consensus.cc:2781] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.511476 13243 ts_tablet_manager.cc:1909] T d843fe82af1148f4a9045ce8b09ad11d P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.515058 13124 raft_consensus.cc:2454] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:56:36.516552 13436 raft_consensus.cc:686] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:36.517964 13249 tablet_bootstrap.cc:654] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.517912 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89eb4e2701894f32a85d2cd89da58ec2" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:36.518482 13252 leader_election.cc:288] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.518714 12783 leader_election.cc:302] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:36.517145 13436 consensus_queue.cc:237] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.521466 13252 raft_consensus.cc:2781] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.521755 12987 raft_consensus.cc:3037] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.526120 12987 raft_consensus.cc:2454] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:56:36.526293 13245 tablet_bootstrap.cc:492] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.526460 13245 ts_tablet_manager.cc:1367] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.003s
I20240506 19:56:36.527333 13245 raft_consensus.cc:348] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.527568 13245 raft_consensus.cc:374] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.527666 13245 raft_consensus.cc:729] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.527995 13517 raft_consensus.cc:482] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.528561 13245 consensus_queue.cc:260] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.529541 13244 tablet_bootstrap.cc:492] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.529716 13244 ts_tablet_manager.cc:1367] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.265s user 0.026s sys 0.059s
I20240506 19:56:36.530069 13245 ts_tablet_manager.cc:1398] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:36.530429 13245 tablet_bootstrap.cc:492] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.530355 13517 raft_consensus.cc:504] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.531741 13517 leader_election.cc:288] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.532083 13245 tablet_bootstrap.cc:654] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.532302 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe1c6ed89d643ecb4e97402d78167b6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.532951 13124 raft_consensus.cc:2454] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:36.530812 13244 raft_consensus.cc:348] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.533473 13244 raft_consensus.cc:374] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.533573 13244 raft_consensus.cc:729] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.533800 13244 consensus_queue.cc:260] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.534361 13244 ts_tablet_manager.cc:1398] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:36.534687 13244 tablet_bootstrap.cc:492] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.535413 12783 leader_election.cc:302] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:36.537590 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17fb091762144f6591152680a6acd29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.539670 13244 tablet_bootstrap.cc:654] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.540964 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe1c6ed89d643ecb4e97402d78167b6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:36.541379 12987 raft_consensus.cc:2454] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:36.544328 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4095f9fc28ff48f68064ec8318ff7b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:36.554706 13242 ts_tablet_manager.cc:1875] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.558075 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d843fe82af1148f4a9045ce8b09ad11d (table default.Table_bbc623ea167c48929e77cee2b3d07aaa [id=f6f5f08248a142d6929736e7872ae027]) successfully deleted
I20240506 19:56:36.558815 13243 tablet_replica.cc:330] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:36.559660 13243 raft_consensus.cc:2229] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.560002 13243 raft_consensus.cc:2258] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.559976 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 104337875517445f862d82ece3a84889
I20240506 19:56:36.560583 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 104337875517445f862d82ece3a84889 deleted and replaced by tablet 176c61208b1c4518820a829d9f5c0a6f
I20240506 19:56:36.560863 13243 ts_tablet_manager.cc:1875] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.564158 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 104337875517445f862d82ece3a84889 with delete_type TABLET_DATA_DELETED (replaced by tablet 176c61208b1c4518820a829d9f5c0a6f) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:36.565737 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 104337875517445f862d82ece3a84889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104337875517445f862d82ece3a84889 already in progress: creating tablet
I20240506 19:56:36.566285 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 104337875517445f862d82ece3a84889 with delete_type TABLET_DATA_DELETED (replaced by tablet 176c61208b1c4518820a829d9f5c0a6f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.567960 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4be929f0e72e4d22a9af2fe6b3072bbe
I20240506 19:56:36.568558 12519 catalog_manager.cc:6554] ReplaceTablet: tablet 4be929f0e72e4d22a9af2fe6b3072bbe deleted and replaced by tablet 2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:36.569010 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe with delete_type TABLET_DATA_DELETED (replaced by tablet 2f7f7171d06948388bc0fdb3514202ed) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.569052 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ca6325447fde4d198454d0ac20e928bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.569851 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca6325447fde4d198454d0ac20e928bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.570183 13249 tablet_bootstrap.cc:492] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.570338 13249 ts_tablet_manager.cc:1367] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.004s
W20240506 19:56:36.570389 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4be929f0e72e4d22a9af2fe6b3072bbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be929f0e72e4d22a9af2fe6b3072bbe already in progress: creating tablet
I20240506 19:56:36.571218 13249 raft_consensus.cc:348] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.571427 13249 raft_consensus.cc:374] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.571513 13249 raft_consensus.cc:729] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.571799 13249 consensus_queue.cc:260] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.574644 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe with delete_type TABLET_DATA_DELETED (replaced by tablet 2f7f7171d06948388bc0fdb3514202ed) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.580715 13249 ts_tablet_manager.cc:1398] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240506 19:56:36.582098 13249 tablet_bootstrap.cc:492] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.586910 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_59add447dae84c509690e4f1f689fb40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.587553 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59add447dae84c509690e4f1f689fb40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.589972 13249 tablet_bootstrap.cc:654] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.593127 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a36b4d51b100486c831a41a52d81fd66
I20240506 19:56:36.593683 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 104337875517445f862d82ece3a84889 with delete_type TABLET_DATA_DELETED (replaced by tablet 176c61208b1c4518820a829d9f5c0a6f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.594993 13243 ts_tablet_manager.cc:1888] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.595463 13517 raft_consensus.cc:2781] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.595420 12522 catalog_manager.cc:5484] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:36.597613 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet a36b4d51b100486c831a41a52d81fd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:36.598654 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a36b4d51b100486c831a41a52d81fd66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a36b4d51b100486c831a41a52d81fd66 already in progress: creating tablet
I20240506 19:56:36.599370 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet a36b4d51b100486c831a41a52d81fd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.604162 13245 tablet_bootstrap.cc:492] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.604373 13245 ts_tablet_manager.cc:1367] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.074s user 0.024s sys 0.015s
I20240506 19:56:36.604763 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe with delete_type TABLET_DATA_DELETED (replaced by tablet 2f7f7171d06948388bc0fdb3514202ed) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.605450 13245 raft_consensus.cc:348] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.605715 13245 raft_consensus.cc:374] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.605804 13245 raft_consensus.cc:729] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.606204 13245 consensus_queue.cc:260] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.606928 13245 ts_tablet_manager.cc:1398] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:36.607084 13243 log.cc:1196] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/8596757ef75e433fb801f7679db5fefb
I20240506 19:56:36.606499 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet a36b4d51b100486c831a41a52d81fd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.607780 13245 tablet_bootstrap.cc:492] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.611824 13517 raft_consensus.cc:482] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.613613 13245 tablet_bootstrap.cc:654] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.615240 13243 ts_tablet_manager.cc:1909] T 8596757ef75e433fb801f7679db5fefb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.616838 13517 raft_consensus.cc:3037] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.618103 13244 tablet_bootstrap.cc:492] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.618299 13243 tablet_replica.cc:330] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:36.618480 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 8596757ef75e433fb801f7679db5fefb (table default.Table_73b96b599455444fa8fead40db334a12 [id=a76c2b5f25a24c55a8379608a9857eda]) successfully deleted
I20240506 19:56:36.618538 13243 raft_consensus.cc:2229] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.618723 13243 raft_consensus.cc:2258] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.619439 13243 ts_tablet_manager.cc:1875] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.619860 13244 ts_tablet_manager.cc:1367] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.031s
I20240506 19:56:36.620766 13252 raft_consensus.cc:686] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:56:36.621389 13244 raft_consensus.cc:348] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.621927 13244 raft_consensus.cc:374] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.622025 13244 raft_consensus.cc:729] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.622707 13244 consensus_queue.cc:260] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.623323 13244 ts_tablet_manager.cc:1398] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:36.624760 13244 tablet_bootstrap.cc:492] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.626042 13252 consensus_queue.cc:237] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:36.631362 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b consensus state has no leader
I20240506 19:56:36.631881 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cc8e3aa90fea4998b3e0a4a411add544"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.632475 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc8e3aa90fea4998b3e0a4a411add544 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.637934 13244 tablet_bootstrap.cc:654] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.638757 13249 tablet_bootstrap.cc:492] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.639024 13249 ts_tablet_manager.cc:1367] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.006s
I20240506 19:56:36.639106 13243 ts_tablet_manager.cc:1888] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.639281 13243 log.cc:1196] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/30cb3cb7cbe14cc9bf1bfd07227f538e
I20240506 19:56:36.639616 13243 ts_tablet_manager.cc:1909] T 30cb3cb7cbe14cc9bf1bfd07227f538e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.640604 13243 tablet_replica.cc:330] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:36.640705 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 30cb3cb7cbe14cc9bf1bfd07227f538e (table default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc [id=8b4499b016334533ab4366c2a4b1bd72]) successfully deleted
I20240506 19:56:36.640810 13243 raft_consensus.cc:2229] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.640977 13243 raft_consensus.cc:2258] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.641541 13243 ts_tablet_manager.cc:1875] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.641891 13249 raft_consensus.cc:348] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.642403 13249 raft_consensus.cc:374] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.642539 13249 raft_consensus.cc:729] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.642838 13249 consensus_queue.cc:260] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.643512 13249 ts_tablet_manager.cc:1398] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:36.643882 13249 tablet_bootstrap.cc:492] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.643926 13245 tablet_bootstrap.cc:492] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.647090 13245 ts_tablet_manager.cc:1367] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.006s
I20240506 19:56:36.648080 13245 raft_consensus.cc:348] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.648350 13245 raft_consensus.cc:374] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.648444 13245 raft_consensus.cc:729] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.648744 13245 consensus_queue.cc:260] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.652001 13245 ts_tablet_manager.cc:1398] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:36.653041 13243 ts_tablet_manager.cc:1888] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.653172 13243 log.cc:1196] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d0e3260f46024386baac64cb8677883c
I20240506 19:56:36.653458 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 17fb091762144f6591152680a6acd29e
I20240506 19:56:36.653604 13243 ts_tablet_manager.cc:1909] T d0e3260f46024386baac64cb8677883c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.654415 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_085ebc849e674a89a8b03e33327a342d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:36.654686 13245 tablet_bootstrap.cc:492] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:36.655040 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_085ebc849e674a89a8b03e33327a342d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.656293 13243 tablet_replica.cc:330] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:36.656491 13243 raft_consensus.cc:2229] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.656669 13243 raft_consensus.cc:2258] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.656807 13245 tablet_bootstrap.cc:654] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.656884 13436 raft_consensus.cc:482] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.657047 13436 raft_consensus.cc:504] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.657572 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d0e3260f46024386baac64cb8677883c (table default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]) successfully deleted
I20240506 19:56:36.657720 13436 leader_election.cc:288] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:36.657860 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2751074a54ed4bcea6979e6ad11c2efe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2751074a54ed4bcea6979e6ad11c2efe already in progress: creating tablet
I20240506 19:56:36.658246 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fb091762144f6591152680a6acd29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.658782 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fb091762144f6591152680a6acd29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:36.658834 12922 leader_election.cc:341] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8842523a97954e3f81ffd87530201d93
I20240506 19:56:36.659417 13245 tablet_bootstrap.cc:492] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.659432 13243 ts_tablet_manager.cc:1875] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.659569 13245 ts_tablet_manager.cc:1367] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
W20240506 19:56:36.659868 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 17fb091762144f6591152680a6acd29e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17fb091762144f6591152680a6acd29e already in progress: creating tablet
I20240506 19:56:36.661339 13245 raft_consensus.cc:348] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.661548 13245 raft_consensus.cc:374] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.661619 13245 raft_consensus.cc:729] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.661877 13245 consensus_queue.cc:260] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.662612 13245 ts_tablet_manager.cc:1398] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 19:56:36.665920 13245 tablet_bootstrap.cc:492] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.667668 13245 tablet_bootstrap.cc:654] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.669761 12521 catalog_manager.cc:5484] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:36.670419 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_62b69a40ec6a432186ff79fc5693659e [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 5 got 0
I20240506 19:56:36.671788 13249 tablet_bootstrap.cc:654] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:36.672509 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6156c6f582b747de973b357b6d071a46 consensus state has no leader
I20240506 19:56:36.673056 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_feb73e349178483fb087f43a269605b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.673753 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feb73e349178483fb087f43a269605b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.674866 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8842523a97954e3f81ffd87530201d93" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.675109 12847 raft_consensus.cc:2454] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:36.675733 12921 leader_election.cc:302] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:36.675906 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 81ea09efd4aa4019911dd23774f464a2
I20240506 19:56:36.676195 13436 raft_consensus.cc:2781] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.676297 13436 raft_consensus.cc:482] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.676399 13436 raft_consensus.cc:3037] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.676841 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_37e9aef6398b47059569bab0a7265a94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.677493 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37e9aef6398b47059569bab0a7265a94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.677932 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ea09efd4aa4019911dd23774f464a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.678223 13246 tablet_replica.cc:330] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.678448 13246 raft_consensus.cc:2229] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.678640 13246 raft_consensus.cc:2258] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.679116 13436 raft_consensus.cc:504] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.679203 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ea09efd4aa4019911dd23774f464a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.679980 13433 raft_consensus.cc:482] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.680100 13433 raft_consensus.cc:504] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.680683 13246 ts_tablet_manager.cc:1875] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.680723 13433 leader_election.cc:288] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.681327 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c18888ae3d6420ca1de3b13fc36cf6c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.681562 13124 raft_consensus.cc:2454] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:36.682111 12922 leader_election.cc:302] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:36.683180 12922 leader_election.cc:341] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8842523a97954e3f81ffd87530201d93
I20240506 19:56:36.683458 13436 leader_election.cc:288] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.683601 13436 raft_consensus.cc:2781] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.684020 13436 raft_consensus.cc:482] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.684245 13436 raft_consensus.cc:3037] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.686282 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:36.687294 13436 raft_consensus.cc:504] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.687258 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f0dfb5ea8de346149cc9b1f245fd94aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:36.687438 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:36.687891 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0dfb5ea8de346149cc9b1f245fd94aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.689167 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c18888ae3d6420ca1de3b13fc36cf6c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.689421 13124 raft_consensus.cc:3037] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.690580 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.692852 13124 raft_consensus.cc:2454] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:36.693491 12922 leader_election.cc:302] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:36.693823 13433 raft_consensus.cc:2781] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.694010 13433 raft_consensus.cc:686] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:36.694284 13433 consensus_queue.cc:237] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.697942 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fb091762144f6591152680a6acd29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.699067 13246 ts_tablet_manager.cc:1888] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:36.700174 13436 leader_election.cc:288] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.701007 13246 log.cc:1196] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/81ea09efd4aa4019911dd23774f464a2
I20240506 19:56:36.701973 13246 ts_tablet_manager.cc:1909] T 81ea09efd4aa4019911dd23774f464a2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.703100 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.704386 13249 tablet_bootstrap.cc:492] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.704532 13249 ts_tablet_manager.cc:1367] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.003s
I20240506 19:56:36.707008 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ea09efd4aa4019911dd23774f464a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.707185 13243 ts_tablet_manager.cc:1888] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.707350 13243 log.cc:1196] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/fbe1c6ed89d643ecb4e97402d78167b6
I20240506 19:56:36.707741 13243 ts_tablet_manager.cc:1909] T fbe1c6ed89d643ecb4e97402d78167b6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.708001 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8842523a97954e3f81ffd87530201d93" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.708258 12847 raft_consensus.cc:3037] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.709826 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c18888ae3d6420ca1de3b13fc36cf6c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:36.710441 12921 leader_election.cc:341] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.711536 13329 raft_consensus.cc:482] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.711745 13329 raft_consensus.cc:504] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.711941 13245 tablet_bootstrap.cc:492] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.712093 13245 ts_tablet_manager.cc:1367] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.011s
I20240506 19:56:36.710836 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c18888ae3d6420ca1de3b13fc36cf6c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.712917 13245 raft_consensus.cc:348] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.713131 13245 raft_consensus.cc:374] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.713208 13329 leader_election.cc:288] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.713810 13517 raft_consensus.cc:504] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.713958 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:36.713989 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b7e42061cd487b8517447eb59ea8a1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.714882 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f89d91417e7c4a6e9ede8f6c8548ee81" } new_table_name: "default.Table_24bc90d17d404ca3884c13c02624e651" modify_external_catalogs: true
W20240506 19:56:36.715051 13058 leader_election.cc:341] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.715204 13517 leader_election.cc:288] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:36.715392 12921 leader_election.cc:341] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.713233 13245 raft_consensus.cc:729] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.716408 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b7e42061cd487b8517447eb59ea8a1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:36.716619 12988 raft_consensus.cc:2454] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:36.716629 13245 consensus_queue.cc:260] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.716881 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.717586 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.718050 13057 leader_election.cc:302] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.718789 13329 raft_consensus.cc:2781] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.718875 13329 raft_consensus.cc:482] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.718967 13329 raft_consensus.cc:3037] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.719471 13245 ts_tablet_manager.cc:1398] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:36.719918 13245 tablet_bootstrap.cc:492] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.721083 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:36.722198 13244 tablet_bootstrap.cc:492] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.722342 13244 ts_tablet_manager.cc:1367] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.023s
W20240506 19:56:36.723105 12781 leader_election.cc:341] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: fbe1c6ed89d643ecb4e97402d78167b6
I20240506 19:56:36.723187 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]) successfully deleted
I20240506 19:56:36.724086 13329 raft_consensus.cc:504] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.724377 13244 raft_consensus.cc:348] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.724607 13244 raft_consensus.cc:374] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.724740 13244 raft_consensus.cc:729] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:36.724773 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 618c56ef126a4ca48b6a379a6771bc1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 618c56ef126a4ca48b6a379a6771bc1c already in progress: creating tablet
I20240506 19:56:36.724978 13244 consensus_queue.cc:260] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.725430 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b7e42061cd487b8517447eb59ea8a1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.725589 13244 ts_tablet_manager.cc:1398] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20240506 19:56:36.725744 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ca394b7f53ce445e8778dd25d0cf25c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca394b7f53ce445e8778dd25d0cf25c7 already in progress: creating tablet
I20240506 19:56:36.725917 13244 tablet_bootstrap.cc:492] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:36.727526 13058 leader_election.cc:341] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.727973 13244 tablet_bootstrap.cc:654] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.728539 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b7e42061cd487b8517447eb59ea8a1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:36.728799 12988 raft_consensus.cc:3037] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.733031 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe1c6ed89d643ecb4e97402d78167b6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.733392 13329 leader_election.cc:288] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:36.733659 13124 raft_consensus.cc:3037] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.733770 13245 tablet_bootstrap.cc:654] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.732815 12988 raft_consensus.cc:2454] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:36.737721 13057 leader_election.cc:302] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.738197 13124 raft_consensus.cc:2454] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:56:36.738698 13329 raft_consensus.cc:2781] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.738739 12783 leader_election.cc:302] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745
I20240506 19:56:36.739113 13329 raft_consensus.cc:686] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:36.739296 13517 raft_consensus.cc:2781] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.739456 13517 raft_consensus.cc:686] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:56:36.739521 13329 consensus_queue.cc:237] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.739760 13517 consensus_queue.cc:237] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.740692 13436 raft_consensus.cc:482] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.740842 13436 raft_consensus.cc:504] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.741869 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80834187f874661b611b43065eee943" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:36.742435 12921 leader_election.cc:341] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.743352 13436 leader_election.cc:288] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.743795 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80834187f874661b611b43065eee943" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.744120 13124 raft_consensus.cc:2454] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:36.744618 12922 leader_election.cc:302] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.748410 12847 raft_consensus.cc:2454] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:36.749045 12921 leader_election.cc:302] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:36.749526 13353 raft_consensus.cc:2781] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.749682 13353 raft_consensus.cc:482] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.749780 13353 raft_consensus.cc:3037] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.753134 13353 raft_consensus.cc:504] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.754109 13353 leader_election.cc:288] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.754283 13353 raft_consensus.cc:2781] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.754387 13353 raft_consensus.cc:686] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:36.754642 13353 consensus_queue.cc:237] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.729790 13249 raft_consensus.cc:348] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.755947 13249 raft_consensus.cc:374] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.756045 13249 raft_consensus.cc:729] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.756772 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80834187f874661b611b43065eee943" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.756994 13249 consensus_queue.cc:260] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:36.757436 12921 leader_election.cc:341] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.763284 13242 ts_tablet_manager.cc:1888] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.763513 13242 log.cc:1196] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/09856d2dd66043fbab08966d04e7b992
I20240506 19:56:36.764087 13242 ts_tablet_manager.cc:1909] T 09856d2dd66043fbab08966d04e7b992 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.764436 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80834187f874661b611b43065eee943" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.764726 13124 raft_consensus.cc:3037] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.767694 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1e4ddadc6b5a4ce9810916112217bff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:36.768352 13436 raft_consensus.cc:482] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:36.768440 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e4ddadc6b5a4ce9810916112217bff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.768688 13436 raft_consensus.cc:504] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.774137 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581aae0889654903933bef3bf1026954" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:36.774753 12922 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.774988 13436 leader_election.cc:288] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.770095 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581aae0889654903933bef3bf1026954" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:36.779845 12921 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.779976 12921 leader_election.cc:302] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.781029 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 09856d2dd66043fbab08966d04e7b992 (table default.Table_0f2e3be5e4274da5ac5270fced5e3d84 [id=4021900d99ce4302b2de9b33a687e94f]) successfully deleted
W20240506 19:56:36.781752 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2751074a54ed4bcea6979e6ad11c2efe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2751074a54ed4bcea6979e6ad11c2efe already in progress: creating tablet
I20240506 19:56:36.782102 13436 raft_consensus.cc:2726] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:36.789254 12988 raft_consensus.cc:1261] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Refusing update from remote peer 8a9c37db9d4f41cd93820a1c81f48d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:36.790688 13500 consensus_queue.cc:1035] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:36.793047 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f68de13c23f14d00b8f74290c11fcde3" } new_table_name: "default.Table_8be586b55bf2436698bfeb2f94fb465d" modify_external_catalogs: true
I20240506 19:56:36.795336 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet da41ef03182445f1946584c2bf015139 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:36.798858 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.799110 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet e34e128329204418b7d6fb55762fb590 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.799326 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:36.800981 13532 log.cc:924] Time spent T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240506 19:56:36.812944 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 89eb4e2701894f32a85d2cd89da58ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.814095 13246 tablet_replica.cc:330] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.814463 13246 raft_consensus.cc:2229] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.814798 13246 raft_consensus.cc:2258] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.815699 13246 ts_tablet_manager.cc:1875] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.821045 13124 raft_consensus.cc:2454] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:36.822150 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0e2005ef58fd4fcea16abe771ee385f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e2005ef58fd4fcea16abe771ee385f8 already in progress: creating tablet
I20240506 19:56:36.823464 13243 tablet_replica.cc:330] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:36.823529 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet fbe1c6ed89d643ecb4e97402d78167b6 (table default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]) successfully deleted
W20240506 19:56:36.827270 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 07755e37a20646a3b1a7b900d54a077c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07755e37a20646a3b1a7b900d54a077c already in progress: creating tablet
I20240506 19:56:36.828640 13244 tablet_bootstrap.cc:492] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.828845 13244 ts_tablet_manager.cc:1367] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.005s
I20240506 19:56:36.828900 12847 raft_consensus.cc:1261] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Refusing update from remote peer 8a9c37db9d4f41cd93820a1c81f48d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:36.829566 13500 consensus_queue.cc:1035] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:36.829710 13244 raft_consensus.cc:348] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.829949 13244 raft_consensus.cc:374] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.830024 13244 raft_consensus.cc:729] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.832089 13244 consensus_queue.cc:260] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.832626 13244 ts_tablet_manager.cc:1398] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:36.832929 13244 tablet_bootstrap.cc:492] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.834601 13244 tablet_bootstrap.cc:654] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:36.844060 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet e17197adc6ca4593ac3fc494548a6e67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e17197adc6ca4593ac3fc494548a6e67 already in progress: creating tablet
W20240506 19:56:36.844942 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ea25fb8e42234b82885b8fb2275d4925 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea25fb8e42234b82885b8fb2275d4925 already in progress: creating tablet
I20240506 19:56:36.826205 12922 leader_election.cc:302] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:36.847059 13249 ts_tablet_manager.cc:1398] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.142s user 0.001s sys 0.006s
W20240506 19:56:36.849758 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet fdab4110236e4f5c8690f6d963ff27f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdab4110236e4f5c8690f6d963ff27f7 already in progress: creating tablet
W20240506 19:56:36.853545 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 581aae0889654903933bef3bf1026954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 581aae0889654903933bef3bf1026954 already in progress: creating tablet
I20240506 19:56:36.854300 13244 tablet_bootstrap.cc:492] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.854482 13244 ts_tablet_manager.cc:1367] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240506 19:56:36.855440 13244 raft_consensus.cc:348] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.855712 13244 raft_consensus.cc:374] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.855844 13244 raft_consensus.cc:729] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.856843 13245 tablet_bootstrap.cc:492] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.857033 13245 ts_tablet_manager.cc:1367] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.019s
I20240506 19:56:36.858270 13245 raft_consensus.cc:348] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.858491 13245 raft_consensus.cc:374] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.858577 13245 raft_consensus.cc:729] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.859340 13245 consensus_queue.cc:260] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.860040 13244 consensus_queue.cc:260] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.860694 13244 ts_tablet_manager.cc:1398] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:36.861002 13244 tablet_bootstrap.cc:492] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.862811 13245 ts_tablet_manager.cc:1398] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 19:56:36.863209 13245 tablet_bootstrap.cc:492] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.865034 13245 tablet_bootstrap.cc:654] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.865793 13242 tablet_replica.cc:330] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:36.866046 13242 raft_consensus.cc:2229] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:36.866551 13242 pending_rounds.cc:62] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Trying to abort 1 pending ops.
W20240506 19:56:36.866858 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d8f3e7df09f347bfa5cad4de34dbda13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8f3e7df09f347bfa5cad4de34dbda13 already in progress: creating tablet
W20240506 19:56:36.867264 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0a8b12f36e74c95ad6eaf30e432f3e9 already in progress: creating tablet
I20240506 19:56:36.847319 13243 raft_consensus.cc:2229] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.859943 13546 tablet.cc:1722] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:36.868665 13538 raft_consensus.cc:2781] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.869222 13556 tablet.cc:1722] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:36.872613 13249 tablet_bootstrap.cc:492] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.875092 13538 raft_consensus.cc:686] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:36.875638 13243 raft_consensus.cc:2258] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.877135 13243 ts_tablet_manager.cc:1875] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.876485 13538 consensus_queue.cc:237] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.880937 13242 pending_rounds.cc:68] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744025047859200 op_type: NO_OP noop_request { }
I20240506 19:56:36.881178 13242 raft_consensus.cc:2866] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:36.881341 13242 raft_consensus.cc:2258] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.883107 13549 tablet.cc:1722] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:36.885931 13242 ts_tablet_manager.cc:1875] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.868541 13244 tablet_bootstrap.cc:654] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:36.892053 12922 consensus_peers.cc:485] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ea09efd4aa4019911dd23774f464a2. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:36.875905 13551 tablet.cc:1722] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:36.897356 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]). Expected version 1 got 0
I20240506 19:56:36.898368 13245 tablet_bootstrap.cc:492] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.898526 13245 ts_tablet_manager.cc:1367] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240506 19:56:36.899379 13245 raft_consensus.cc:348] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.899552 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet a80834187f874661b611b43065eee943 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.899608 13245 raft_consensus.cc:374] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.899693 13245 raft_consensus.cc:729] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.899838 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e3260f46024386baac64cb8677883c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.900101 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet a16704567ef6454eaef886fb9cd599ce with delete_type TABLET_DATA_DELETED (replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:36.902972 13249 tablet_bootstrap.cc:654] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.904703 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet a80834187f874661b611b43065eee943 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.905117 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 09856d2dd66043fbab08966d04e7b992 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d04a1c5ee7b4d529fa2f4da4945ff09) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:36.907856 13245 consensus_queue.cc:260] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.909054 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b0e5bbcbcb324f54bc4feab7c750c029" } new_table_name: "default.Table_a7a0fc1296414d58a1cc512c170d3e96" modify_external_catalogs: true
I20240506 19:56:36.911162 13245 ts_tablet_manager.cc:1398] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240506 19:56:36.911159 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_08345b91e36e40d7860b550554026c61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:36.911764 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08345b91e36e40d7860b550554026c61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:36.912046 13246 ts_tablet_manager.cc:1888] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.912246 13246 log.cc:1196] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/fbe1c6ed89d643ecb4e97402d78167b6
I20240506 19:56:36.912853 13246 ts_tablet_manager.cc:1909] T fbe1c6ed89d643ecb4e97402d78167b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.912958 13245 tablet_bootstrap.cc:492] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.897341 13244 tablet_bootstrap.cc:492] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.913338 13244 ts_tablet_manager.cc:1367] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240506 19:56:36.914276 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet fbe1c6ed89d643ecb4e97402d78167b6 (table default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]) successfully deleted
I20240506 19:56:36.914649 13246 tablet_replica.cc:330] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.914923 13246 raft_consensus.cc:2229] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:36.914891 13244 raft_consensus.cc:348] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.915187 13246 raft_consensus.cc:2258] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.915208 13244 raft_consensus.cc:374] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.915331 13244 raft_consensus.cc:729] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.915742 13244 consensus_queue.cc:260] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.919154 13244 ts_tablet_manager.cc:1398] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240506 19:56:36.919637 13244 tablet_bootstrap.cc:492] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.932828 13571 tablet.cc:1722] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:36.934773 13246 ts_tablet_manager.cc:1875] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.935549 13572 tablet.cc:1722] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:36.936878 13245 tablet_bootstrap.cc:654] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.940445 13244 tablet_bootstrap.cc:654] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.946166 13249 tablet_bootstrap.cc:492] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.946378 13249 ts_tablet_manager.cc:1367] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.002s
I20240506 19:56:36.947178 13249 raft_consensus.cc:348] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.947497 13249 raft_consensus.cc:374] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.947597 13249 raft_consensus.cc:729] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.947908 13249 consensus_queue.cc:260] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.948577 13249 ts_tablet_manager.cc:1398] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:36.948921 13249 tablet_bootstrap.cc:492] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:36.954080 13538 raft_consensus.cc:482] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.954279 13538 raft_consensus.cc:504] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.955404 13538 leader_election.cc:288] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.955740 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7f56fea34946e590dae0621b28f9c6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:36.956322 13124 raft_consensus.cc:2454] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:36.957006 12922 leader_election.cc:302] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:36.957278 13538 raft_consensus.cc:2781] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:36.957398 13538 raft_consensus.cc:482] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:36.957515 13538 raft_consensus.cc:3037] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.960368 13538 raft_consensus.cc:504] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.960774 13245 tablet_bootstrap.cc:492] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:36.961037 13245 ts_tablet_manager.cc:1367] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20240506 19:56:36.961681 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7f56fea34946e590dae0621b28f9c6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:36.962000 13124 raft_consensus.cc:3037] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:36.962100 13245 raft_consensus.cc:348] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.962430 13245 raft_consensus.cc:374] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.962524 13245 raft_consensus.cc:729] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:36.965023 13124 raft_consensus.cc:2454] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:36.965610 13538 leader_election.cc:288] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:36.965695 12922 leader_election.cc:302] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:36.966271 13538 raft_consensus.cc:2781] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:36.966459 13538 raft_consensus.cc:686] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:36.966758 13538 consensus_queue.cc:237] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.968887 13245 consensus_queue.cc:260] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:36.971275 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7f56fea34946e590dae0621b28f9c6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:36.971515 13245 ts_tablet_manager.cc:1398] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240506 19:56:36.973318 13244 tablet_bootstrap.cc:492] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:36.973490 13244 ts_tablet_manager.cc:1367] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.006s
W20240506 19:56:36.973937 12921 leader_election.cc:341] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.974288 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fe3e87d2202147518431c995667b07fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:36.974488 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7f56fea34946e590dae0621b28f9c6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:36.974444 13244 raft_consensus.cc:348] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:36.974686 13244 raft_consensus.cc:374] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:36.974799 13244 raft_consensus.cc:729] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:36.974941 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe3e87d2202147518431c995667b07fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:36.975155 12921 leader_election.cc:341] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:36.975145 13244 consensus_queue.cc:260] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:36.976325 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ba2365d00044c32a06f9017a207adf7 consensus state has no leader
I20240506 19:56:36.976615 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet a8b3ed0b56aa478abb44347afca8fcf6 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:36.977380 13244 ts_tablet_manager.cc:1398] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:36.977437 13249 tablet_bootstrap.cc:654] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.977784 13244 tablet_bootstrap.cc:492] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:36.979135 13242 ts_tablet_manager.cc:1888] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:36.980154 13245 tablet_bootstrap.cc:492] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:36.981122 13242 log.cc:1196] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/fbe1c6ed89d643ecb4e97402d78167b6
I20240506 19:56:36.981870 13242 ts_tablet_manager.cc:1909] T fbe1c6ed89d643ecb4e97402d78167b6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.982245 13245 tablet_bootstrap.cc:654] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.970528 13246 ts_tablet_manager.cc:1888] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.985607 13246 log.cc:1196] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/e34e128329204418b7d6fb55762fb590
I20240506 19:56:36.986011 13246 ts_tablet_manager.cc:1909] T e34e128329204418b7d6fb55762fb590 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:36.986891 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet fbe1c6ed89d643ecb4e97402d78167b6 (table default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]) successfully deleted
I20240506 19:56:36.987241 13246 tablet_replica.cc:330] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:36.987449 13242 tablet_replica.cc:330] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:36.987409 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet e34e128329204418b7d6fb55762fb590 (table default.Table_f44dcd3362d64342ad02f7d17b3f24d0 [id=a147e46c173c402db02793b1e11de5b5]) successfully deleted
I20240506 19:56:36.987500 13246 raft_consensus.cc:2229] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:36.987735 13242 raft_consensus.cc:2229] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:36.987839 13246 pending_rounds.cc:62] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:36.987926 13246 pending_rounds.cc:68] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744025047986176 op_type: NO_OP noop_request { }
I20240506 19:56:36.988143 13242 pending_rounds.cc:62] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Trying to abort 1 pending ops.
I20240506 19:56:36.988257 13242 pending_rounds.cc:68] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744024634077184 op_type: NO_OP noop_request { }
I20240506 19:56:36.988452 13547 raft_consensus.cc:482] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:36.988606 13242 raft_consensus.cc:2866] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:36.988763 13242 raft_consensus.cc:2258] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:36.989743 13242 ts_tablet_manager.cc:1875] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:36.994038 13244 tablet_bootstrap.cc:654] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:36.997870 13243 ts_tablet_manager.cc:1888] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:36.998020 13243 log.cc:1196] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a223c33ba56c4ddbba46fedbce660ae2
I20240506 19:56:36.998405 13243 ts_tablet_manager.cc:1909] T a223c33ba56c4ddbba46fedbce660ae2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:36.998778 13249 tablet_bootstrap.cc:492] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:36.998991 13249 ts_tablet_manager.cc:1367] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.002s
I20240506 19:56:36.999271 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 216) took 1051 ms. Trace:
I20240506 19:56:36.999419 13243 rpcz_store.cc:276] 0506 19:56:35.948269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:35.948339 (+ 70us) service_pool.cc:225] Handling call
0506 19:56:36.823359 (+875020us) ts_tablet_manager.cc:1120] Deleting tablet a223c33ba56c4ddbba46fedbce660ae2
0506 19:56:36.823369 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:36.906660 (+ 83291us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:36.997714 (+ 91054us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:36.999224 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":293,"tablet-delete.queue_time_us":874789}
I20240506 19:56:36.999892 13243 tablet_replica.cc:330] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:37.000079 13242 ts_tablet_manager.cc:1888] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:37.000211 13242 log.cc:1196] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/89eb4e2701894f32a85d2cd89da58ec2
I20240506 19:56:37.000386 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a223c33ba56c4ddbba46fedbce660ae2 (table default.Table_218a52f567cf4031bee1b7834387ac84 [id=779ea3288b6d40ceb9636fdde3697472]) successfully deleted
I20240506 19:56:37.000591 13242 ts_tablet_manager.cc:1909] T 89eb4e2701894f32a85d2cd89da58ec2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:36.988606 13547 raft_consensus.cc:504] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.002081 13547 leader_election.cc:288] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.001430 13246 raft_consensus.cc:2866] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:37.002764 13246 raft_consensus.cc:2258] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.005489 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:37.006453 13547 raft_consensus.cc:482] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:37.006529 13058 leader_election.cc:341] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:37.006654 13547 raft_consensus.cc:504] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.007336 13547 leader_election.cc:288] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.007795 13246 ts_tablet_manager.cc:1875] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.008903 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e3260f46024386baac64cb8677883c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:37.009122 12847 raft_consensus.cc:2454] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:37.009575 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 89eb4e2701894f32a85d2cd89da58ec2 (table default.Table_dae4e46e818c4147960f5d606ce3b969 [id=098471374f5e4282b0a6a64b15b4f9ea]) successfully deleted
I20240506 19:56:37.009651 13058 leader_election.cc:302] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:37.010169 13547 raft_consensus.cc:2781] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:37.010280 13547 raft_consensus.cc:482] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:37.010397 13547 raft_consensus.cc:3037] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:37.010962 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 104337875517445f862d82ece3a84889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104337875517445f862d82ece3a84889 already in progress: creating tablet
W20240506 19:56:37.011731 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4be929f0e72e4d22a9af2fe6b3072bbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be929f0e72e4d22a9af2fe6b3072bbe already in progress: creating tablet
W20240506 19:56:37.012498 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a36b4d51b100486c831a41a52d81fd66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a36b4d51b100486c831a41a52d81fd66 already in progress: creating tablet
W20240506 19:56:37.013375 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 17fb091762144f6591152680a6acd29e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17fb091762144f6591152680a6acd29e already in progress: creating tablet
I20240506 19:56:37.013432 13547 raft_consensus.cc:504] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:37.014281 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 618c56ef126a4ca48b6a379a6771bc1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 618c56ef126a4ca48b6a379a6771bc1c already in progress: creating tablet
I20240506 19:56:37.014338 13244 tablet_bootstrap.cc:492] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.014523 13244 ts_tablet_manager.cc:1367] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.001s
I20240506 19:56:37.014469 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e3260f46024386baac64cb8677883c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:37.014707 12847 raft_consensus.cc:3037] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.014878 13242 tablet_replica.cc:330] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:37.015064 13242 raft_consensus.cc:2229] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.006465 13245 tablet_bootstrap.cc:492] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:37.015466 13244 raft_consensus.cc:348] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.015698 13245 ts_tablet_manager.cc:1367] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20240506 19:56:37.015744 13244 raft_consensus.cc:374] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.015849 13244 raft_consensus.cc:729] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.016268 13244 consensus_queue.cc:260] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.016415 13245 raft_consensus.cc:348] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.016594 13245 raft_consensus.cc:374] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.016882 13245 raft_consensus.cc:729] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.017146 13245 consensus_queue.cc:260] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.017565 13547 leader_election.cc:288] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.019212 13244 ts_tablet_manager.cc:1398] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240506 19:56:37.019553 13244 tablet_bootstrap.cc:492] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.020198 13245 ts_tablet_manager.cc:1398] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:37.021353 13245 tablet_bootstrap.cc:492] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:37.021962 13244 tablet_bootstrap.cc:654] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.022895 13245 tablet_bootstrap.cc:654] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.015846 13242 raft_consensus.cc:2258] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.023553 13242 ts_tablet_manager.cc:1875] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.025012 13246 ts_tablet_manager.cc:1888] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:37.025213 13246 log.cc:1196] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:37.025800 13246 ts_tablet_manager.cc:1909] T 64b7e42061cd487b8517447eb59ea8a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:37.027614 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 64b7e42061cd487b8517447eb59ea8a1 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]) successfully deleted
I20240506 19:56:37.028067 13246 tablet_replica.cc:330] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:37.028308 13246 raft_consensus.cc:2229] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.028579 13246 raft_consensus.cc:2258] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.029182 13246 ts_tablet_manager.cc:1875] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.031118 12847 raft_consensus.cc:2454] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:37.031735 13058 leader_election.cc:302] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:37.032019 13547 raft_consensus.cc:2781] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:37.032167 13547 raft_consensus.cc:686] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:37.032455 13547 consensus_queue.cc:237] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.034133 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 1667a0bbd3af4d678434b068c47cc518 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.036428 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet da41ef03182445f1946584c2bf015139
I20240506 19:56:37.037025 12519 catalog_manager.cc:6554] ReplaceTablet: tablet da41ef03182445f1946584c2bf015139 deleted and replaced by tablet 472c63d8995346d78633154a8f0d4c2b
I20240506 19:56:37.037353 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet da41ef03182445f1946584c2bf015139 with delete_type TABLET_DATA_DELETED (replaced by tablet 472c63d8995346d78633154a8f0d4c2b) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:37.037746 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3ec577dd06640068575288c24e1105a consensus state has no leader
I20240506 19:56:37.038084 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b7f56fea34946e590dae0621b28f9c6 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.039505 13244 tablet_bootstrap.cc:492] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.039711 13244 ts_tablet_manager.cc:1367] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240506 19:56:37.040774 13244 raft_consensus.cc:348] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.041054 13244 raft_consensus.cc:374] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.041167 13244 raft_consensus.cc:729] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.041565 13244 consensus_queue.cc:260] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:37.043287 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 5988b89cb52e4f858b52862138bed025 (table default.Table_ac80e9e0789c4cbaab8826eb3c0bc621 [id=cd2382012b444dd09cd9ad1a50a20b4c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5988b89cb52e4f858b52862138bed025 consensus state has no leader
I20240506 19:56:37.044929 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet da41ef03182445f1946584c2bf015139 with delete_type TABLET_DATA_DELETED (replaced by tablet 472c63d8995346d78633154a8f0d4c2b) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.045634 13243 raft_consensus.cc:2229] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.045926 13243 raft_consensus.cc:2258] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.046945 13244 ts_tablet_manager.cc:1398] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240506 19:56:37.051297 13244 tablet_bootstrap.cc:492] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.053172 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet da41ef03182445f1946584c2bf015139 with delete_type TABLET_DATA_DELETED (replaced by tablet 472c63d8995346d78633154a8f0d4c2b) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:37.053364 13057 leader_election.cc:341] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d0e3260f46024386baac64cb8677883c
I20240506 19:56:37.053478 13243 ts_tablet_manager.cc:1875] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:37.053797 13057 leader_election.cc:341] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d0e3260f46024386baac64cb8677883c
W20240506 19:56:37.055650 12921 consensus_peers.cc:485] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:37.056138 13242 ts_tablet_manager.cc:1888] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:37.056304 13242 log.cc:1196] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/81ea09efd4aa4019911dd23774f464a2
I20240506 19:56:37.056875 13242 ts_tablet_manager.cc:1909] T 81ea09efd4aa4019911dd23774f464a2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:37.057152 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:37.057391 12988 raft_consensus.cc:2454] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:37.057826 13124 raft_consensus.cc:1261] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:37.057929 13057 leader_election.cc:302] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:37.058234 13500 raft_consensus.cc:2781] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:37.058329 13500 raft_consensus.cc:482] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:37.058430 13500 raft_consensus.cc:3037] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.059119 12847 raft_consensus.cc:3037] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.062088 13500 raft_consensus.cc:504] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.063062 13500 leader_election.cc:288] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.063233 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:37.063645 13433 consensus_queue.cc:1035] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20240506 19:56:37.064775 13058 leader_election.cc:341] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:37.066483 12847 raft_consensus.cc:1261] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:37.070086 13433 consensus_queue.cc:1035] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240506 19:56:37.073874 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:37.074148 12988 raft_consensus.cc:3037] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.074757 13244 tablet_bootstrap.cc:654] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.076951 12988 raft_consensus.cc:2454] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:37.076934 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]) successfully deleted
I20240506 19:56:37.077526 13057 leader_election.cc:302] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:37.077860 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ca394b7f53ce445e8778dd25d0cf25c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca394b7f53ce445e8778dd25d0cf25c7 already in progress: creating tablet
I20240506 19:56:37.078998 13500 raft_consensus.cc:2781] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
W20240506 19:56:37.079218 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet da41ef03182445f1946584c2bf015139 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da41ef03182445f1946584c2bf015139 already in progress: creating tablet
I20240506 19:56:37.079303 13500 raft_consensus.cc:686] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:37.080060 13500 consensus_queue.cc:237] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.081415 13244 tablet_bootstrap.cc:492] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.081629 13244 ts_tablet_manager.cc:1367] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.002s
I20240506 19:56:37.082139 13249 raft_consensus.cc:348] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.082389 13249 raft_consensus.cc:374] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.082476 13249 raft_consensus.cc:729] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.082460 13244 raft_consensus.cc:348] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.082801 13244 raft_consensus.cc:374] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.082754 13249 consensus_queue.cc:260] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.082890 13244 raft_consensus.cc:729] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.083143 13244 consensus_queue.cc:260] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.083642 13244 ts_tablet_manager.cc:1398] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:37.083971 13244 tablet_bootstrap.cc:492] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.085556 13244 tablet_bootstrap.cc:654] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.087577 13249 ts_tablet_manager.cc:1398] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.088s user 0.001s sys 0.001s
I20240506 19:56:37.091285 13243 ts_tablet_manager.cc:1888] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:37.091495 13243 log.cc:1196] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/89eb4e2701894f32a85d2cd89da58ec2
I20240506 19:56:37.091904 13243 ts_tablet_manager.cc:1909] T 89eb4e2701894f32a85d2cd89da58ec2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:37.092207 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.253:36819 every 8 connection(s)
I20240506 19:56:37.093030 13313 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.253:36819
I20240506 19:56:37.094389 13244 tablet_bootstrap.cc:492] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.095388 13313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-1/data/info.pb
I20240506 19:56:37.095645 13244 ts_tablet_manager.cc:1367] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20240506 19:56:37.097132 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 13313
I20240506 19:56:37.097994 13244 raft_consensus.cc:348] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.098239 13244 raft_consensus.cc:374] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.098351 13244 raft_consensus.cc:729] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.100980 13244 consensus_queue.cc:260] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.101805 13244 ts_tablet_manager.cc:1398] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:37.102312 13244 tablet_bootstrap.cc:492] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.105126 13244 tablet_bootstrap.cc:654] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.108692 13249 tablet_bootstrap.cc:492] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:37.111044 13249 tablet_bootstrap.cc:654] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.117698 13244 tablet_bootstrap.cc:492] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.118000 13244 ts_tablet_manager.cc:1367] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20240506 19:56:37.118943 13244 raft_consensus.cc:348] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.119272 13244 raft_consensus.cc:374] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.119405 13244 raft_consensus.cc:729] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.119740 13244 consensus_queue.cc:260] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.120457 13244 ts_tablet_manager.cc:1398] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:37.120851 13244 tablet_bootstrap.cc:492] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.121414 13572 tablet.cc:1722] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240506 19:56:37.122315 13546 tablet.cc:1722] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240506 19:56:37.122723 13244 tablet_bootstrap.cc:654] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.135001 13246 ts_tablet_manager.cc:1888] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:37.135191 13246 log.cc:1196] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/89eb4e2701894f32a85d2cd89da58ec2
I20240506 19:56:37.135669 13246 ts_tablet_manager.cc:1909] T 89eb4e2701894f32a85d2cd89da58ec2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:37.135941 13245 tablet_bootstrap.cc:492] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:37.136118 13245 ts_tablet_manager.cc:1367] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.115s user 0.001s sys 0.003s
I20240506 19:56:37.136644 13244 tablet_bootstrap.cc:492] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.136854 13244 ts_tablet_manager.cc:1367] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240506 19:56:37.137094 13245 raft_consensus.cc:348] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.137351 13245 raft_consensus.cc:374] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.137432 13245 raft_consensus.cc:729] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.137727 13244 raft_consensus.cc:348] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.138077 13244 raft_consensus.cc:374] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.138190 13244 raft_consensus.cc:729] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.138442 13244 consensus_queue.cc:260] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.138628 13245 consensus_queue.cc:260] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.139118 13245 ts_tablet_manager.cc:1398] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:37.139175 13244 ts_tablet_manager.cc:1398] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:37.139459 13244 tablet_bootstrap.cc:492] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.139459 13245 tablet_bootstrap.cc:492] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:37.141182 13245 tablet_bootstrap.cc:654] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.141208 13244 tablet_bootstrap.cc:654] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.145915 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d23f0a5ff6d430282256e9bddc186b9 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.151058 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4b69070c2c254336bfaec6ea640736d0" } new_table_name: "default.Table_b2887df9eaee49beba57959a096c0c65" modify_external_catalogs: true
I20240506 19:56:37.152842 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bb529dd8d37f40449f27b0e07cbc93dc" } modify_external_catalogs: true
I20240506 19:56:37.153041 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bb529dd8d37f40449f27b0e07cbc93dc" } modify_external_catalogs: true
I20240506 19:56:37.157961 13246 tablet_replica.cc:330] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:37.158361 13246 raft_consensus.cc:2229] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.158610 13246 raft_consensus.cc:2258] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.162245 13547 raft_consensus.cc:482] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.162451 13547 raft_consensus.cc:504] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.163336 13547 leader_election.cc:288] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.164072 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16704567ef6454eaef886fb9cd599ce" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:37.164345 12846 raft_consensus.cc:2454] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:37.164347 13057 leader_election.cc:341] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:37.166662 13058 leader_election.cc:302] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745
I20240506 19:56:37.167742 13249 tablet_bootstrap.cc:492] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:37.168008 13249 ts_tablet_manager.cc:1367] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.005s
I20240506 19:56:37.171144 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fe77d7d403a041b4b9f6a1c97177f461" } modify_external_catalogs: true
I20240506 19:56:37.171376 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e3428b42c94e4d8ca3620c86ea4d3082" } new_table_name: "default.Table_95be5bd2c51b4613b0152034aae6350d" modify_external_catalogs: true
I20240506 19:56:37.171404 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fe77d7d403a041b4b9f6a1c97177f461" } modify_external_catalogs: true
I20240506 19:56:37.172724 13547 raft_consensus.cc:2781] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:37.172866 13547 raft_consensus.cc:482] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:37.172964 13547 raft_consensus.cc:3037] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.173395 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 217) took 1143 ms. Trace:
I20240506 19:56:37.173504 13243 rpcz_store.cc:276] 0506 19:56:35.949065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:35.949129 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:36.999834 (+1050705us) ts_tablet_manager.cc:1120] Deleting tablet 89eb4e2701894f32a85d2cd89da58ec2
0506 19:56:36.999842 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.076041 (+ 76199us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:37.091146 (+ 15105us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:37.092744 (+ 1598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050468}
I20240506 19:56:37.173837 13551 tablet.cc:1722] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240506 19:56:37.174006 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 89eb4e2701894f32a85d2cd89da58ec2 (table default.Table_dae4e46e818c4147960f5d606ce3b969 [id=098471374f5e4282b0a6a64b15b4f9ea]) successfully deleted
I20240506 19:56:37.174376 13243 tablet_replica.cc:330] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:37.174559 13243 raft_consensus.cc:2229] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.174738 13243 raft_consensus.cc:2258] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.175271 13243 ts_tablet_manager.cc:1875] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.176434 13547 raft_consensus.cc:504] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.169087 13249 raft_consensus.cc:348] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.177585 13249 raft_consensus.cc:374] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.177682 13249 raft_consensus.cc:729] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.178040 13246 ts_tablet_manager.cc:1875] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.177970 13249 consensus_queue.cc:260] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.178588 13249 ts_tablet_manager.cc:1398] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:37.181653 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16704567ef6454eaef886fb9cd599ce" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:37.181929 12846 raft_consensus.cc:3037] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.183977 13249 tablet_bootstrap.cc:492] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:37.186471 13249 tablet_bootstrap.cc:654] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.189604 12846 raft_consensus.cc:2454] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:37.195394 13058 leader_election.cc:302] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:37.195770 13500 raft_consensus.cc:2781] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:37.195910 13500 raft_consensus.cc:686] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:37.196188 13500 consensus_queue.cc:237] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.198194 13547 leader_election.cc:288] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:37.199048 13057 leader_election.cc:341] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:37.205163 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 (DEFAULT_TABLE table=default.Table_56eb06f51079472894642f592bbe43a3 [id=577dd46fb1b44fd380574a6cd0ba2695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.205817 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6e0126d9854215a0e174a76c0a0391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.206298 12520 catalog_manager.cc:5484] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:37.206784 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:37.207023 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 03e896b3ba7c4864a26cdfce76fadcfb (table default.Table_fe77d7d403a041b4b9f6a1c97177f461 [id=15e39c71182c4a089e9c950679153305]). Expected version 1 got 0
I20240506 19:56:37.207235 12520 catalog_manager.cc:5484] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:37.207636 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]). Expected version 1 got 0
I20240506 19:56:37.210743 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c (DEFAULT_TABLE table=default.Table_a410467878c2481f86118d12ccd06f28 [id=f339c5ce6df04d839431b2475a6c1d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.211323 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb90bf767c94d73ace6411cfd31d65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.213732 12819 tablet_service.cc:1467] Processing CreateTablet for tablet b3e50bde3564493eb7267acb42cb8592 (DEFAULT_TABLE table=default.Table_e3428b42c94e4d8ca3620c86ea4d3082 [id=b8f3e6ecf02f4dd299adfa1d7464e53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.214249 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e50bde3564493eb7267acb42cb8592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.214316 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 (DEFAULT_TABLE table=default.Table_56eb06f51079472894642f592bbe43a3 [id=577dd46fb1b44fd380574a6cd0ba2695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.214732 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6e0126d9854215a0e174a76c0a0391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.217650 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c (DEFAULT_TABLE table=default.Table_a410467878c2481f86118d12ccd06f28 [id=f339c5ce6df04d839431b2475a6c1d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.217784 12965 tablet_service.cc:1467] Processing CreateTablet for tablet b3e50bde3564493eb7267acb42cb8592 (DEFAULT_TABLE table=default.Table_e3428b42c94e4d8ca3620c86ea4d3082 [id=b8f3e6ecf02f4dd299adfa1d7464e53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.218051 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb90bf767c94d73ace6411cfd31d65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.218247 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e50bde3564493eb7267acb42cb8592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.218916 13035 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:37.222846 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 (DEFAULT_TABLE table=default.Table_53b38bd9c00f46f2abac3c4ca498fb0a [id=7a2c513cd8e74817b4a391c9c7932085]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.223112 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 (DEFAULT_TABLE table=default.Table_53b38bd9c00f46f2abac3c4ca498fb0a [id=7a2c513cd8e74817b4a391c9c7932085]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.223261 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e8b8be953a4a5784f2d098fd6364e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.223631 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e8b8be953a4a5784f2d098fd6364e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.223821 12821 tablet_service.cc:1467] Processing CreateTablet for tablet bafd3c4706f74e7da36a8f3538760f43 (DEFAULT_TABLE table=default.Table_162d0c1d88d04a11b0015afaea52e19f [id=5a2afe178f674b29834ccf87ec1ed218]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.224264 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafd3c4706f74e7da36a8f3538760f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.224545 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 176c61208b1c4518820a829d9f5c0a6f (DEFAULT_TABLE table=default.Table_4b69070c2c254336bfaec6ea640736d0 [id=54457b4758cf4a5b8f7a55ef6b21f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.224916 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176c61208b1c4518820a829d9f5c0a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.225585 12962 tablet_service.cc:1467] Processing CreateTablet for tablet be6e733b6efd4041abf465c7bb788ef5 (DEFAULT_TABLE table=default.Table_ba47b7bc355a4589a2b439289c6a61de [id=c6e62c9299cf403ba333d67e1dee2dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.225941 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6e733b6efd4041abf465c7bb788ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.226104 12826 tablet_service.cc:1467] Processing CreateTablet for tablet be6e733b6efd4041abf465c7bb788ef5 (DEFAULT_TABLE table=default.Table_ba47b7bc355a4589a2b439289c6a61de [id=c6e62c9299cf403ba333d67e1dee2dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.226497 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6e733b6efd4041abf465c7bb788ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.224267 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e (DEFAULT_TABLE table=default.Table_c3e795b5a9724754a33f79ebe9ac2c1f [id=343720fdaa3d43b28031433246aebe9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.229588 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7f7171d06948388bc0fdb3514202ed (DEFAULT_TABLE table=default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.229975 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7f7171d06948388bc0fdb3514202ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.231348 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7f7171d06948388bc0fdb3514202ed (DEFAULT_TABLE table=default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.233606 12958 tablet_service.cc:1467] Processing CreateTablet for tablet b899fb38bfed4e02b3708f6abec80e0a (DEFAULT_TABLE table=default.Table_d6d662a23cfb4d02be80346ea62fbe2a [id=8dca79d6b027426c8f12402280fea23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.233999 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b899fb38bfed4e02b3708f6abec80e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.234505 12816 tablet_service.cc:1467] Processing CreateTablet for tablet b899fb38bfed4e02b3708f6abec80e0a (DEFAULT_TABLE table=default.Table_d6d662a23cfb4d02be80346ea62fbe2a [id=8dca79d6b027426c8f12402280fea23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.235592 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 304a1401627948bbaff78fae4d42ab22 (DEFAULT_TABLE table=default.Table_788bbfde3ac54d13b047fa3c065680d0 [id=3a715f69316d4a67b261118ff0069c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.236454 12824 tablet_service.cc:1467] Processing CreateTablet for tablet f8951bf4873e4382bcd42a52368fb376 (DEFAULT_TABLE table=default.Table_6974ae6398ae4559a50fb236133424b1 [id=07a119f6e5854b1090f2f47dc8c9dada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.236778 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 63f56d2180b1472a8168fb3a63589065 (DEFAULT_TABLE table=default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.237119 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 304a1401627948bbaff78fae4d42ab22 (DEFAULT_TABLE table=default.Table_788bbfde3ac54d13b047fa3c065680d0 [id=3a715f69316d4a67b261118ff0069c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.237565 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304a1401627948bbaff78fae4d42ab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.237767 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607b3fc2c17b4793af9b84c19066ec5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.238080 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304a1401627948bbaff78fae4d42ab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.238516 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7f7171d06948388bc0fdb3514202ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.239184 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b899fb38bfed4e02b3708f6abec80e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.239688 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8951bf4873e4382bcd42a52368fb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.240125 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f56d2180b1472a8168fb3a63589065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.240365 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 (DEFAULT_TABLE table=default.Table_56eb06f51079472894642f592bbe43a3 [id=577dd46fb1b44fd380574a6cd0ba2695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.240598 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 89eb4e2701894f32a85d2cd89da58ec2 (table default.Table_dae4e46e818c4147960f5d606ce3b969 [id=098471374f5e4282b0a6a64b15b4f9ea]) successfully deleted
I20240506 19:56:37.240816 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6e0126d9854215a0e174a76c0a0391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.241647 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c (DEFAULT_TABLE table=default.Table_a410467878c2481f86118d12ccd06f28 [id=f339c5ce6df04d839431b2475a6c1d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.242086 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb90bf767c94d73ace6411cfd31d65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.244904 12963 tablet_service.cc:1467] Processing CreateTablet for tablet bafd3c4706f74e7da36a8f3538760f43 (DEFAULT_TABLE table=default.Table_162d0c1d88d04a11b0015afaea52e19f [id=5a2afe178f674b29834ccf87ec1ed218]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.245584 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 63f56d2180b1472a8168fb3a63589065 (DEFAULT_TABLE table=default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.245620 13094 tablet_service.cc:1467] Processing CreateTablet for tablet b3e50bde3564493eb7267acb42cb8592 (DEFAULT_TABLE table=default.Table_e3428b42c94e4d8ca3620c86ea4d3082 [id=b8f3e6ecf02f4dd299adfa1d7464e53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.246042 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e50bde3564493eb7267acb42cb8592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.246085 12961 tablet_service.cc:1467] Processing CreateTablet for tablet f8951bf4873e4382bcd42a52368fb376 (DEFAULT_TABLE table=default.Table_6974ae6398ae4559a50fb236133424b1 [id=07a119f6e5854b1090f2f47dc8c9dada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.246569 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e (DEFAULT_TABLE table=default.Table_c3e795b5a9724754a33f79ebe9ac2c1f [id=343720fdaa3d43b28031433246aebe9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.247045 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 176c61208b1c4518820a829d9f5c0a6f (DEFAULT_TABLE table=default.Table_4b69070c2c254336bfaec6ea640736d0 [id=54457b4758cf4a5b8f7a55ef6b21f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.247138 13246 ts_tablet_manager.cc:1888] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:37.247288 13246 log.cc:1196] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a80834187f874661b611b43065eee943
W20240506 19:56:37.247731 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176c61208b1c4518820a829d9f5c0a6f consensus state has no leader
I20240506 19:56:37.247917 13246 ts_tablet_manager.cc:1909] T a80834187f874661b611b43065eee943 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:37.248180 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b122543bad9446f9bd6b57a8336b17bc" } modify_external_catalogs: true
I20240506 19:56:37.248407 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b122543bad9446f9bd6b57a8336b17bc" } modify_external_catalogs: true
W20240506 19:56:37.249018 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3e50bde3564493eb7267acb42cb8592 consensus state has no leader
I20240506 19:56:37.249691 13246 tablet_replica.cc:330] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:37.249678 13092 tablet_service.cc:1467] Processing CreateTablet for tablet be6e733b6efd4041abf465c7bb788ef5 (DEFAULT_TABLE table=default.Table_ba47b7bc355a4589a2b439289c6a61de [id=c6e62c9299cf403ba333d67e1dee2dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.249925 13246 raft_consensus.cc:2229] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:37.250195 13246 pending_rounds.cc:62] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:37.250228 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7f7171d06948388bc0fdb3514202ed (DEFAULT_TABLE table=default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.250270 13246 pending_rounds.cc:68] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744026247557120 op_type: NO_OP noop_request { }
I20240506 19:56:37.250416 13246 raft_consensus.cc:2866] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:37.250530 13246 raft_consensus.cc:2258] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.250705 13100 tablet_service.cc:1467] Processing CreateTablet for tablet b899fb38bfed4e02b3708f6abec80e0a (DEFAULT_TABLE table=default.Table_d6d662a23cfb4d02be80346ea62fbe2a [id=8dca79d6b027426c8f12402280fea23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.251195 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 304a1401627948bbaff78fae4d42ab22 (DEFAULT_TABLE table=default.Table_788bbfde3ac54d13b047fa3c065680d0 [id=3a715f69316d4a67b261118ff0069c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.251339 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 (DEFAULT_TABLE table=default.Table_53b38bd9c00f46f2abac3c4ca498fb0a [id=7a2c513cd8e74817b4a391c9c7932085]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.251633 13095 tablet_service.cc:1467] Processing CreateTablet for tablet bafd3c4706f74e7da36a8f3538760f43 (DEFAULT_TABLE table=default.Table_162d0c1d88d04a11b0015afaea52e19f [id=5a2afe178f674b29834ccf87ec1ed218]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.251879 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e24bec7e49144632ab0da2f0ae398f9a" } modify_external_catalogs: true
I20240506 19:56:37.252075 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 63f56d2180b1472a8168fb3a63589065 (DEFAULT_TABLE table=default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.252085 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e24bec7e49144632ab0da2f0ae398f9a" } modify_external_catalogs: true
I20240506 19:56:37.252553 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafd3c4706f74e7da36a8f3538760f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.252539 13099 tablet_service.cc:1467] Processing CreateTablet for tablet f8951bf4873e4382bcd42a52368fb376 (DEFAULT_TABLE table=default.Table_6974ae6398ae4559a50fb236133424b1 [id=07a119f6e5854b1090f2f47dc8c9dada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.252794 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607b3fc2c17b4793af9b84c19066ec5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.253460 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8951bf4873e4382bcd42a52368fb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.257458 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176c61208b1c4518820a829d9f5c0a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.257918 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e (DEFAULT_TABLE table=default.Table_c3e795b5a9724754a33f79ebe9ac2c1f [id=343720fdaa3d43b28031433246aebe9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.258884 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 176c61208b1c4518820a829d9f5c0a6f (DEFAULT_TABLE table=default.Table_4b69070c2c254336bfaec6ea640736d0 [id=54457b4758cf4a5b8f7a55ef6b21f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.261456 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f56d2180b1472a8168fb3a63589065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.262029 13244 tablet_bootstrap.cc:492] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.262244 13244 ts_tablet_manager.cc:1367] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.009s
I20240506 19:56:37.263234 13244 raft_consensus.cc:348] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.263491 13244 raft_consensus.cc:374] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.263579 13244 raft_consensus.cc:729] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.263883 13244 consensus_queue.cc:260] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.264446 13244 ts_tablet_manager.cc:1398] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:37.265663 13244 tablet_bootstrap.cc:492] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.268092 13244 tablet_bootstrap.cc:654] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.269135 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6e733b6efd4041abf465c7bb788ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.269774 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607b3fc2c17b4793af9b84c19066ec5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.270234 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176c61208b1c4518820a829d9f5c0a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.270679 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafd3c4706f74e7da36a8f3538760f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.271122 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e8b8be953a4a5784f2d098fd6364e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.271515 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8951bf4873e4382bcd42a52368fb376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.271926 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b899fb38bfed4e02b3708f6abec80e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.272317 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f56d2180b1472a8168fb3a63589065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.272720 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7f7171d06948388bc0fdb3514202ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.273123 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304a1401627948bbaff78fae4d42ab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.277565 13249 tablet_bootstrap.cc:492] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:37.281466 13249 ts_tablet_manager.cc:1367] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20240506 19:56:37.282385 13249 raft_consensus.cc:348] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.282629 13249 raft_consensus.cc:374] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.282738 13249 raft_consensus.cc:729] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.282991 13249 consensus_queue.cc:260] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.288580 13249 ts_tablet_manager.cc:1398] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:37.288929 13249 tablet_bootstrap.cc:492] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:37.290750 13249 tablet_bootstrap.cc:654] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.300725 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a80834187f874661b611b43065eee943 (table default.Table_2b51a8ac571347a896b8741066c330c4 [id=c18e335476d443f9949f5f38fa4550ae]) successfully deleted
I20240506 19:56:37.323280 12846 raft_consensus.cc:3037] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.328317 13433 consensus_queue.cc:1035] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:37.334513 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a8b3ed0b56aa478abb44347afca8fcf6
I20240506 19:56:37.334925 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4a080ccac6394210b0b24b65843f9d75
I20240506 19:56:37.335023 12518 catalog_manager.cc:6554] ReplaceTablet: tablet a8b3ed0b56aa478abb44347afca8fcf6 deleted and replaced by tablet 8eb9296bb1104b4cbe279eecd3683796
I20240506 19:56:37.335984 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:37.336373 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a080ccac6394210b0b24b65843f9d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.337072 12513 catalog_manager.cc:6554] ReplaceTablet: tablet 7d23f0a5ff6d430282256e9bddc186b9 deleted and replaced by tablet 99f4861a01404663870ce91978c8af4f
I20240506 19:56:37.337191 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 99f4861a01404663870ce91978c8af4f) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:37.337787 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4a080ccac6394210b0b24b65843f9d75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a080ccac6394210b0b24b65843f9d75 already in progress: creating tablet
W20240506 19:56:37.338434 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7d23f0a5ff6d430282256e9bddc186b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d23f0a5ff6d430282256e9bddc186b9 already in progress: creating tablet
I20240506 19:56:37.339179 12515 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:37.339380 12515 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_92eb5ef0ea1c408c98a0a14387f76b6e [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 6 got 0
I20240506 19:56:37.340948 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 8842523a97954e3f81ffd87530201d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.341250 13242 tablet_replica.cc:330] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:37.341506 13242 raft_consensus.cc:2229] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.341691 13242 raft_consensus.cc:2258] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.342322 13242 ts_tablet_manager.cc:1875] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.343155 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 99f4861a01404663870ce91978c8af4f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.343513 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a080ccac6394210b0b24b65843f9d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.343715 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9296bb1104b4cbe279eecd3683796) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.345433 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe1c6ed89d643ecb4e97402d78167b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 72a3bed0eb234e21b890242e9d23a5b4) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.346220 13517 raft_consensus.cc:482] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.346388 13517 raft_consensus.cc:504] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.347095 13517 leader_election.cc:288] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:37.348630 12781 leader_election.cc:341] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a223c33ba56c4ddbba46fedbce660ae2
I20240506 19:56:37.349882 13242 ts_tablet_manager.cc:1888] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:37.350025 13242 log.cc:1196] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/8842523a97954e3f81ffd87530201d93
I20240506 19:56:37.350351 13242 ts_tablet_manager.cc:1909] T 8842523a97954e3f81ffd87530201d93 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:37.351691 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 8842523a97954e3f81ffd87530201d93 (table default.Table_0594be752cc441b1af65142e6d925488 [id=b1277bb384294e7ea3cd5d0a52b5c813]) successfully deleted
I20240506 19:56:37.351800 12895 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
W20240506 19:56:37.351838 12783 leader_election.cc:341] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a223c33ba56c4ddbba46fedbce660ae2
W20240506 19:56:37.351977 12507 catalog_manager.cc:4797] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet fbe1c6ed89d643ecb4e97402d78167b6 because the tablet was not found. No further retry: Not found: Tablet not found: fbe1c6ed89d643ecb4e97402d78167b6
I20240506 19:56:37.351965 12783 leader_election.cc:302] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:37.352420 13517 raft_consensus.cc:2726] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:37.354135 13433 raft_consensus.cc:482] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.354341 13433 raft_consensus.cc:504] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.353426 13547 consensus_queue.cc:1035] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240506 19:56:37.355078 13433 leader_election.cc:288] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:37.355489 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9296bb1104b4cbe279eecd3683796) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.357441 13171 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
W20240506 19:56:37.358439 12922 leader_election.cc:341] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 09856d2dd66043fbab08966d04e7b992
I20240506 19:56:37.358868 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d6d662a23cfb4d02be80346ea62fbe2a" } modify_external_catalogs: true
I20240506 19:56:37.359074 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d6d662a23cfb4d02be80346ea62fbe2a" } modify_external_catalogs: true
I20240506 19:56:37.359731 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f29103e88f944dbf883f0de41a88c053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.360356 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f29103e88f944dbf883f0de41a88c053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.361420 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_58c604462fcb4b9cab62687d64877b02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.362147 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58c604462fcb4b9cab62687d64877b02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.364595 12848 raft_consensus.cc:3037] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.365087 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7f4701bc5da447f58c2c1dad63a0df55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.365630 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f4701bc5da447f58c2c1dad63a0df55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.365664 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6b562516d9b845789bf9d901c015e37c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.366142 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b562516d9b845789bf9d901c015e37c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.366900 12921 leader_election.cc:341] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Not found: Tablet not found: 09856d2dd66043fbab08966d04e7b992
I20240506 19:56:37.367053 12921 leader_election.cc:302] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:37.367497 13433 raft_consensus.cc:2726] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:37.367563 12848 raft_consensus.cc:1261] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Refusing update from remote peer 8a9c37db9d4f41cd93820a1c81f48d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:37.368589 13500 consensus_queue.cc:1035] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:37.378007 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 99f4861a01404663870ce91978c8af4f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.377171 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a080ccac6394210b0b24b65843f9d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.382669 13538 consensus_queue.cc:1035] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:37.385951 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 03e896b3ba7c4864a26cdfce76fadcfb
I20240506 19:56:37.387621 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_46d49a82b855486f994526543440ace7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.388249 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46d49a82b855486f994526543440ace7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.389685 13244 tablet_bootstrap.cc:492] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.389858 13244 ts_tablet_manager.cc:1367] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.011s
I20240506 19:56:37.389796 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_278e4f7429df43d9b9fa9ec82b1d91de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:37.389902 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d909ad6f09e24952918c9471609c9d27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.390337 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_278e4f7429df43d9b9fa9ec82b1d91de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.390399 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d909ad6f09e24952918c9471609c9d27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.390689 13244 raft_consensus.cc:348] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.390906 13244 raft_consensus.cc:374] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.391006 13244 raft_consensus.cc:729] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.391996 13244 consensus_queue.cc:260] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.392764 13244 ts_tablet_manager.cc:1398] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 19:56:37.393077 13244 tablet_bootstrap.cc:492] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.393270 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b058641e17b2474785e2066bd9781690" } new_table_name: "default.Table_82ef5c37628b4923b2e9cce9d2ff4e88" modify_external_catalogs: true
I20240506 19:56:37.395012 13244 tablet_bootstrap.cc:654] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.395468 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6d22301588d54dcd8649bc08a1ade8f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.396067 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d22301588d54dcd8649bc08a1ade8f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.397629 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_adbb55d8d99f4266a048d9a935d73195" } new_table_name: "default.Table_60d08e70820949809d66466222b3742d" modify_external_catalogs: true
I20240506 19:56:37.399060 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 971db6ac24d54efca95695ed1c90d4ed (DEFAULT_TABLE table=default.Table_cc8e3aa90fea4998b3e0a4a411add544 [id=86b683f4dec545118e95b30b194e1fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.399343 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e896b3ba7c4864a26cdfce76fadcfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.399515 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971db6ac24d54efca95695ed1c90d4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.400107 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 5751124adb5b40f59e8d79a4605ae8c4 (DEFAULT_TABLE table=default.Table_08345b91e36e40d7860b550554026c61 [id=2c295d2600344234889c7ad8019bd164]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.400687 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 (DEFAULT_TABLE table=default.Table_feb73e349178483fb087f43a269605b0 [id=b19563cb49834a49924bb9d06b60b6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.401104 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb5a12521044102ae55ca953f1496f9 (DEFAULT_TABLE table=default.Table_085ebc849e674a89a8b03e33327a342d [id=ba90b3aba0c346559318504ce789c81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.401577 12815 tablet_service.cc:1467] Processing CreateTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 (DEFAULT_TABLE table=default.Table_ca6325447fde4d198454d0ac20e928bd [id=a5b2fb638e5a4ee2abfd610cb6274767]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.401633 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (DEFAULT_TABLE table=default.Table_1e4ddadc6b5a4ce9810916112217bff2 [id=16aa3036ed744751afd08515bb792ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.402050 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 9f47449a5a504f939e131743d94433ee (DEFAULT_TABLE table=default.Table_f0dfb5ea8de346149cc9b1f245fd94aa [id=6fcb7b163e714ce7bf9e991838356b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.402602 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 6b59ad067c274fddbba5f71e89480dec (DEFAULT_TABLE table=default.Table_37e9aef6398b47059569bab0a7265a94 [id=d8c76c1f826d49c4818b1e278d407b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.403017 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 472c63d8995346d78633154a8f0d4c2b (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.403362 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 07c7c7b2da9e403f8241a512e210c4d9 (DEFAULT_TABLE table=default.Table_59add447dae84c509690e4f1f689fb40 [id=6fd679db05504418b8f453a92ecf8397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.406090 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9296bb1104b4cbe279eecd3683796) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.407279 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5751124adb5b40f59e8d79a4605ae8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.410176 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2c81ac92d04ddc93cc52e5c01cbc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.411363 13538 raft_consensus.cc:482] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.412676 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 320e714ed6b64f2d84770e7182e4103a (table default.Table_8be586b55bf2436698bfeb2f94fb465d [id=090dfa9462424be1b846e1cb05569156]). Expected version 1 got 0
I20240506 19:56:37.413066 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:37.415779 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]). Expected version 1 got 0
I20240506 19:56:37.418316 13242 tablet_replica.cc:330] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:37.418668 13242 raft_consensus.cc:2229] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.418998 13242 raft_consensus.cc:2258] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.420723 13599 sys_catalog.cc:263] Verifying existing consensus state
I20240506 19:56:37.421355 13517 raft_consensus.cc:482] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.421862 13517 raft_consensus.cc:504] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.422556 13242 ts_tablet_manager.cc:1875] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.422641 13517 leader_election.cc:288] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.422955 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 4acde41b2c03440a89243aae008779e1 (DEFAULT_TABLE table=default.Table_fe3e87d2202147518431c995667b07fc [id=b7788b3408254775a7b73c81911108c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.423637 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acde41b2c03440a89243aae008779e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.424332 13551 tablet.cc:1722] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:37.433614 12895 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:37.443065 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c7c7b2da9e403f8241a512e210c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.449312 13538 raft_consensus.cc:504] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.449807 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f47449a5a504f939e131743d94433ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.451687 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 5751124adb5b40f59e8d79a4605ae8c4 (DEFAULT_TABLE table=default.Table_08345b91e36e40d7860b550554026c61 [id=2c295d2600344234889c7ad8019bd164]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.453187 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 4acde41b2c03440a89243aae008779e1 (DEFAULT_TABLE table=default.Table_fe3e87d2202147518431c995667b07fc [id=b7788b3408254775a7b73c81911108c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.453943 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7f56fea34946e590dae0621b28f9c6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:37.454452 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 07c7c7b2da9e403f8241a512e210c4d9 (DEFAULT_TABLE table=default.Table_59add447dae84c509690e4f1f689fb40 [id=6fd679db05504418b8f453a92ecf8397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.454852 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5751124adb5b40f59e8d79a4605ae8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.455636 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acde41b2c03440a89243aae008779e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.456715 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c7c7b2da9e403f8241a512e210c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.457453 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472c63d8995346d78633154a8f0d4c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.457695 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.457697 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 971db6ac24d54efca95695ed1c90d4ed (DEFAULT_TABLE table=default.Table_cc8e3aa90fea4998b3e0a4a411add544 [id=86b683f4dec545118e95b30b194e1fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.458180 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971db6ac24d54efca95695ed1c90d4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.458356 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb5a12521044102ae55ca953f1496f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.459213 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1615f6167aae4dc096b9f563ab5e6215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.459800 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b59ad067c274fddbba5f71e89480dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.461277 12964 tablet_service.cc:1467] Processing CreateTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 (DEFAULT_TABLE table=default.Table_ca6325447fde4d198454d0ac20e928bd [id=a5b2fb638e5a4ee2abfd610cb6274767]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.461792 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2c81ac92d04ddc93cc52e5c01cbc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.464612 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99dcc418075a4468a1d9a9887c007a20" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:37.466356 12921 leader_election.cc:341] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:37.467772 13546 tablet.cc:1722] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:37.469720 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 (DEFAULT_TABLE table=default.Table_feb73e349178483fb087f43a269605b0 [id=b19563cb49834a49924bb9d06b60b6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.470468 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 92951843c8bd446aa86436b65d9eb793 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.470693 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.469213 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 5751124adb5b40f59e8d79a4605ae8c4 (DEFAULT_TABLE table=default.Table_08345b91e36e40d7860b550554026c61 [id=2c295d2600344234889c7ad8019bd164]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.471459 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7f56fea34946e590dae0621b28f9c6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:37.473799 13245 tablet_bootstrap.cc:492] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:37.473981 13245 ts_tablet_manager.cc:1367] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.335s user 0.003s sys 0.020s
I20240506 19:56:37.474226 13246 ts_tablet_manager.cc:1875] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.474179 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 472c63d8995346d78633154a8f0d4c2b (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.474720 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472c63d8995346d78633154a8f0d4c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.477479 13517 raft_consensus.cc:482] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.477708 13517 raft_consensus.cc:504] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.477878 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet cc8335f9308046279a25a4a95d8e0f65 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.478694 13517 leader_election.cc:288] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.478621 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_54f3105144e741119c67aec80c408fa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.479192 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54f3105144e741119c67aec80c408fa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.480703 12781 leader_election.cc:341] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 8596757ef75e433fb801f7679db5fefb
I20240506 19:56:37.481307 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e896b3ba7c4864a26cdfce76fadcfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.482110 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (DEFAULT_TABLE table=default.Table_1e4ddadc6b5a4ce9810916112217bff2 [id=16aa3036ed744751afd08515bb792ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.482703 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.484426 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 (DEFAULT_TABLE table=default.Table_feb73e349178483fb087f43a269605b0 [id=b19563cb49834a49924bb9d06b60b6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.484786 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1615f6167aae4dc096b9f563ab5e6215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.490368 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 9f47449a5a504f939e131743d94433ee (DEFAULT_TABLE table=default.Table_f0dfb5ea8de346149cc9b1f245fd94aa [id=6fcb7b163e714ce7bf9e991838356b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.490952 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f47449a5a504f939e131743d94433ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.493315 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1615f6167aae4dc096b9f563ab5e6215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.493878 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 6b59ad067c274fddbba5f71e89480dec (DEFAULT_TABLE table=default.Table_37e9aef6398b47059569bab0a7265a94 [id=d8c76c1f826d49c4818b1e278d407b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.494455 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b59ad067c274fddbba5f71e89480dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.497376 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb5a12521044102ae55ca953f1496f9 (DEFAULT_TABLE table=default.Table_085ebc849e674a89a8b03e33327a342d [id=ba90b3aba0c346559318504ce789c81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.497645 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 971db6ac24d54efca95695ed1c90d4ed (DEFAULT_TABLE table=default.Table_cc8e3aa90fea4998b3e0a4a411add544 [id=86b683f4dec545118e95b30b194e1fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.497743 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb5a12521044102ae55ca953f1496f9 (DEFAULT_TABLE table=default.Table_085ebc849e674a89a8b03e33327a342d [id=ba90b3aba0c346559318504ce789c81e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.498252 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb5a12521044102ae55ca953f1496f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:37.502563 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f75467ef60504763a67fc10e8639716c consensus state has no leader
I20240506 19:56:37.502605 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:37.503130 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_01b7228cdeff4166bcd9d1480dbcc479" } new_table_name: "default.Table_11b5bac0d01243f4a527f32ba62d5a38" modify_external_catalogs: true
I20240506 19:56:37.503487 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6b43eacefdd54cc98fa204f64ee70d7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.504061 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b43eacefdd54cc98fa204f64ee70d7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.504807 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_86cbff0af1184af6a66af4ee25521f9d" } new_table_name: "default.Table_cc5c3f3202074991b8db4134d5824ad4" modify_external_catalogs: true
I20240506 19:56:37.505733 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_92eb5ef0ea1c408c98a0a14387f76b6e [id=ee08d3829f304e16b5b8994259bb11b0] alter complete (version 6)
I20240506 19:56:37.506107 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b899fb38bfed4e02b3708f6abec80e0a
I20240506 19:56:37.506987 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet d05c5ca7ff864fa98f3494032729e978 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.510027 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 472c63d8995346d78633154a8f0d4c2b
W20240506 19:56:37.506994 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e2c9f4baf55412d9bc00818a00c652d consensus state has no leader
I20240506 19:56:37.511204 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7f7171d06948388bc0fdb3514202ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.511500 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet b899fb38bfed4e02b3708f6abec80e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.511785 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.512296 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5751124adb5b40f59e8d79a4605ae8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.451313 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 8842523a97954e3f81ffd87530201d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.451398 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 92951843c8bd446aa86436b65d9eb793 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.517809 13123 raft_consensus.cc:2379] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef18e88b87064ed4b6c20f9325ebcc33 in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:56:37.521013 13572 tablet.cc:1722] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240506 19:56:37.523399 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.523521 13551 tablet.cc:1722] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240506 19:56:37.523967 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7f7171d06948388bc0fdb3514202ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.524526 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet b899fb38bfed4e02b3708f6abec80e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.524966 12783 leader_election.cc:302] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:37.525508 13517 raft_consensus.cc:3037] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.528448 13517 raft_consensus.cc:2726] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:37.535442 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb5a12521044102ae55ca953f1496f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.537401 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6" } modify_external_catalogs: true
I20240506 19:56:37.537711 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6" } modify_external_catalogs: true
I20240506 19:56:37.540539 13089 tablet_service.cc:1467] Processing CreateTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 (DEFAULT_TABLE table=default.Table_ca6325447fde4d198454d0ac20e928bd [id=a5b2fb638e5a4ee2abfd610cb6274767]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:37.541508 12922 consensus_peers.cc:485] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8842523a97954e3f81ffd87530201d93. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:37.542014 12783 leader_election.cc:341] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8596757ef75e433fb801f7679db5fefb
I20240506 19:56:37.542189 12783 leader_election.cc:302] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:37.544063 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971db6ac24d54efca95695ed1c90d4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.544931 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 9f47449a5a504f939e131743d94433ee (DEFAULT_TABLE table=default.Table_f0dfb5ea8de346149cc9b1f245fd94aa [id=6fcb7b163e714ce7bf9e991838356b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.545269 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (DEFAULT_TABLE table=default.Table_1e4ddadc6b5a4ce9810916112217bff2 [id=16aa3036ed744751afd08515bb792ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.547497 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d004ba3983954d4aada9c7d0e1a0207d" } new_table_name: "default.Table_f282b6bfb0dd40baa082ac8a5ced166e" modify_external_catalogs: true
I20240506 19:56:37.551281 13245 raft_consensus.cc:348] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.551515 13245 raft_consensus.cc:374] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.552472 13245 raft_consensus.cc:729] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.552755 13245 consensus_queue.cc:260] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.451159 13243 ts_tablet_manager.cc:1888] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:37.553642 13243 log.cc:1196] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:37.554122 13243 ts_tablet_manager.cc:1909] T 64b7e42061cd487b8517447eb59ea8a1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:37.554490 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 218) took 1596 ms. Trace:
I20240506 19:56:37.554646 13243 rpcz_store.cc:276] 0506 19:56:35.958510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:35.958565 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:37.174283 (+1215718us) ts_tablet_manager.cc:1120] Deleting tablet 64b7e42061cd487b8517447eb59ea8a1
0506 19:56:37.174293 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.201921 (+ 27628us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:37.385176 (+183255us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:37.554438 (+169262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215478}
I20240506 19:56:37.555193 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 64b7e42061cd487b8517447eb59ea8a1 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]) successfully deleted
I20240506 19:56:37.555699 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 226) took 1460 ms. Trace:
I20240506 19:56:37.555740 13245 ts_tablet_manager.cc:1398] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.079s user 0.001s sys 0.002s
I20240506 19:56:37.556048 13245 tablet_bootstrap.cc:492] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:37.555951 13243 rpcz_store.cc:276] 0506 19:56:36.095437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.095630 (+ 193us) service_pool.cc:225] Handling call
0506 19:56:37.555508 (+1459878us) ts_tablet_manager.cc:1120] Deleting tablet 0e2005ef58fd4fcea16abe771ee385f8
0506 19:56:37.555521 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.555643 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457669}
W20240506 19:56:37.556602 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0e2005ef58fd4fcea16abe771ee385f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e2005ef58fd4fcea16abe771ee385f8 already in progress: creating tablet
I20240506 19:56:37.556964 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2c81ac92d04ddc93cc52e5c01cbc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.557516 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 227) took 1461 ms. Trace:
I20240506 19:56:37.557641 13243 rpcz_store.cc:276] 0506 19:56:36.095534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.095639 (+ 105us) service_pool.cc:225] Handling call
0506 19:56:37.557337 (+1461698us) ts_tablet_manager.cc:1120] Deleting tablet 07755e37a20646a3b1a7b900d54a077c
0506 19:56:37.557347 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.557470 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454934}
W20240506 19:56:37.558099 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 07755e37a20646a3b1a7b900d54a077c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07755e37a20646a3b1a7b900d54a077c already in progress: creating tablet
I20240506 19:56:37.558480 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 6b59ad067c274fddbba5f71e89480dec (DEFAULT_TABLE table=default.Table_37e9aef6398b47059569bab0a7265a94 [id=d8c76c1f826d49c4818b1e278d407b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.558702 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 228) took 1462 ms. Trace:
I20240506 19:56:37.558841 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.558832 13243 rpcz_store.cc:276] 0506 19:56:36.096001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.096068 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:37.558541 (+1462473us) ts_tablet_manager.cc:1120] Deleting tablet e17197adc6ca4593ac3fc494548a6e67
0506 19:56:37.558551 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.558644 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1453299}
W20240506 19:56:37.559285 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet e17197adc6ca4593ac3fc494548a6e67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e17197adc6ca4593ac3fc494548a6e67 already in progress: creating tablet
I20240506 19:56:37.560133 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 232) took 1419 ms. Trace:
I20240506 19:56:37.560178 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f47449a5a504f939e131743d94433ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.560256 13243 rpcz_store.cc:276] 0506 19:56:36.140280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.140332 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:37.559523 (+1419191us) ts_tablet_manager.cc:1120] Deleting tablet ea25fb8e42234b82885b8fb2275d4925
0506 19:56:37.559532 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.559614 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418426}
W20240506 19:56:37.560772 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ea25fb8e42234b82885b8fb2275d4925 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea25fb8e42234b82885b8fb2275d4925 already in progress: creating tablet
I20240506 19:56:37.561720 13538 leader_election.cc:288] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:37.561720 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e896b3ba7c4864a26cdfce76fadcfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.562422 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 472c63d8995346d78633154a8f0d4c2b (DEFAULT_TABLE table=default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.566044 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 07c7c7b2da9e403f8241a512e210c4d9 (DEFAULT_TABLE table=default.Table_59add447dae84c509690e4f1f689fb40 [id=6fd679db05504418b8f453a92ecf8397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.566856 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b59ad067c274fddbba5f71e89480dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.562083 13547 raft_consensus.cc:482] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.567791 13547 raft_consensus.cc:504] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.568544 13547 leader_election.cc:288] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.569491 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581aae0889654903933bef3bf1026954" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:37.569749 12986 raft_consensus.cc:2454] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:37.570339 13249 tablet_bootstrap.cc:492] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:37.570369 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581aae0889654903933bef3bf1026954" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:37.570823 13249 ts_tablet_manager.cc:1367] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.282s user 0.057s sys 0.065s
I20240506 19:56:37.571890 13249 raft_consensus.cc:348] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.572201 13249 raft_consensus.cc:374] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.572464 13249 raft_consensus.cc:729] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.572814 13249 consensus_queue.cc:260] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.573186 13057 leader_election.cc:302] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:37.573554 13249 ts_tablet_manager.cc:1398] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:37.573626 13244 tablet_bootstrap.cc:492] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:37.573819 13244 ts_tablet_manager.cc:1367] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.181s user 0.007s sys 0.003s
I20240506 19:56:37.574079 13249 tablet_bootstrap.cc:492] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:37.574677 13244 raft_consensus.cc:348] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.574939 13244 raft_consensus.cc:374] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:37.575007 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c7c7b2da9e403f8241a512e210c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.575548 13244 raft_consensus.cc:729] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:37.575610 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99dcc418075a4468a1d9a9887c007a20" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:37.575837 13123 raft_consensus.cc:2454] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:37.575956 13244 consensus_queue.cc:260] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:37.577661 13058 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:37.578233 12922 leader_election.cc:302] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:37.578668 13538 raft_consensus.cc:2781] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:37.578768 13538 raft_consensus.cc:482] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:37.578883 13538 raft_consensus.cc:3037] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.584295 13538 raft_consensus.cc:504] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.585086 13249 tablet_bootstrap.cc:654] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.585727 13538 leader_election.cc:288] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:37.586460 13329 raft_consensus.cc:2781] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:37.586591 13329 raft_consensus.cc:482] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:37.587071 13329 raft_consensus.cc:3037] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.588706 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99dcc418075a4468a1d9a9887c007a20" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:37.588939 12847 raft_consensus.cc:3037] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.590200 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99dcc418075a4468a1d9a9887c007a20" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:37.590415 13123 raft_consensus.cc:3037] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.592088 12847 raft_consensus.cc:2454] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:37.592751 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0c42cb0cd7e349fa86150517492387bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.593495 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c42cb0cd7e349fa86150517492387bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.593511 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fe3e87d2202147518431c995667b07fc" } modify_external_catalogs: true
I20240506 19:56:37.593878 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fe3e87d2202147518431c995667b07fc" } modify_external_catalogs: true
I20240506 19:56:37.595196 13123 raft_consensus.cc:2454] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:37.595741 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 4acde41b2c03440a89243aae008779e1 (DEFAULT_TABLE table=default.Table_fe3e87d2202147518431c995667b07fc [id=b7788b3408254775a7b73c81911108c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.596227 12921 leader_election.cc:302] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:37.597218 13538 raft_consensus.cc:2781] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:37.597546 13538 raft_consensus.cc:686] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:37.597867 13538 consensus_queue.cc:237] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.600801 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 233) took 1458 ms. Trace:
I20240506 19:56:37.600934 13243 rpcz_store.cc:276] 0506 19:56:36.142318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.142368 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:37.600609 (+1458241us) ts_tablet_manager.cc:1120] Deleting tablet fdab4110236e4f5c8690f6d963ff27f7
0506 19:56:37.600622 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.600746 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436515}
I20240506 19:56:37.601399 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 235) took 1426 ms. Trace:
W20240506 19:56:37.601537 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet fdab4110236e4f5c8690f6d963ff27f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdab4110236e4f5c8690f6d963ff27f7 already in progress: creating tablet
I20240506 19:56:37.602924 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7f7171d06948388bc0fdb3514202ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.604270 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet b899fb38bfed4e02b3708f6abec80e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.604627 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.605263 13243 rpcz_store.cc:276] 0506 19:56:36.175011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.175118 (+ 107us) service_pool.cc:225] Handling call
0506 19:56:37.601225 (+1426107us) ts_tablet_manager.cc:1120] Deleting tablet f0a8b12f36e74c95ad6eaf30e432f3e9
0506 19:56:37.601236 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.601364 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420327}
W20240506 19:56:37.605819 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0a8b12f36e74c95ad6eaf30e432f3e9 already in progress: creating tablet
I20240506 19:56:37.605862 13517 raft_consensus.cc:2726] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:37.611480 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acde41b2c03440a89243aae008779e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.614529 13329 raft_consensus.cc:504] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:37.615733 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 236) took 1439 ms. Trace:
W20240506 19:56:37.621702 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3d57249f30a44dab3384092ec20b515 consensus state has no leader
I20240506 19:56:37.622409 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4cbe3de6b961432fbeacafa9ae5fe50c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.623260 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cbe3de6b961432fbeacafa9ae5fe50c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.623284 13243 rpcz_store.cc:276] 0506 19:56:36.175691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.175786 (+ 95us) service_pool.cc:225] Handling call
0506 19:56:37.614095 (+1438309us) ts_tablet_manager.cc:1120] Deleting tablet d8f3e7df09f347bfa5cad4de34dbda13
0506 19:56:37.614392 (+ 297us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.615443 (+ 1051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1430921}
W20240506 19:56:37.626938 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d8f3e7df09f347bfa5cad4de34dbda13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8f3e7df09f347bfa5cad4de34dbda13 already in progress: creating tablet
I20240506 19:56:37.626964 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d05c5ca7ff864fa98f3494032729e978
W20240506 19:56:37.627916 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aae313961a914ed485f866d85785a788 consensus state has no leader
I20240506 19:56:37.628685 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet d05c5ca7ff864fa98f3494032729e978 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.628571 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dfeac1a2083945f18e1355bddf3975ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:37.627924 12518 catalog_manager.cc:6554] ReplaceTablet: tablet d05c5ca7ff864fa98f3494032729e978 deleted and replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c
W20240506 19:56:37.629392 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfeac1a2083945f18e1355bddf3975ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.629812 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cae4c030228d49ff88f3c05320bd8924"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.630653 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cae4c030228d49ff88f3c05320bd8924 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.631135 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b consensus state has no leader
I20240506 19:56:37.634963 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 5988b89cb52e4f858b52862138bed025 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.634570 13243 tablet_replica.cc:330] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:37.640105 13243 raft_consensus.cc:2229] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.642258 13243 raft_consensus.cc:2258] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:37.643395 12921 consensus_peers.cc:485] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8842523a97954e3f81ffd87530201d93. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:37.645120 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Bootstrap starting.
I20240506 19:56:37.645896 13243 ts_tablet_manager.cc:1875] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:37.649241 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet d05c5ca7ff864fa98f3494032729e978 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.653004 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 320e714ed6b64f2d84770e7182e4103a (table default.Table_8be586b55bf2436698bfeb2f94fb465d [id=090dfa9462424be1b846e1cb05569156]). Expected version 1 got 0
I20240506 19:56:37.653772 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb] alter complete (version 4)
I20240506 19:56:37.658457 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e3260f46024386baac64cb8677883c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.659680 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet d843fe82af1148f4a9045ce8b09ad11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.660202 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet a223c33ba56c4ddbba46fedbce660ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.663051 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 8596757ef75e433fb801f7679db5fefb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.663530 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 30cb3cb7cbe14cc9bf1bfd07227f538e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.663956 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b7e42061cd487b8517447eb59ea8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.664422 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 92951843c8bd446aa86436b65d9eb793 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.664882 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet a16704567ef6454eaef886fb9cd599ce with delete_type TABLET_DATA_DELETED (replaced by tablet 20c1590e36b4471d8e72f4b07410ee0f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.665359 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet a80834187f874661b611b43065eee943 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:33 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.669659 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 2c96f014293242be9640515697c0733b (DEFAULT_TABLE table=default.Table_7f4701bc5da447f58c2c1dad63a0df55 [id=f67e8f35a154427f95ae0f771099e941]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.670208 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c96f014293242be9640515697c0733b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.671257 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5988b89cb52e4f858b52862138bed025
I20240506 19:56:37.671416 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4acde41b2c03440a89243aae008779e1
I20240506 19:56:37.671847 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 5988b89cb52e4f858b52862138bed025 deleted and replaced by tablet 4973e600038742a6bec0bb2d8f3551fe
I20240506 19:56:37.672616 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 5988b89cb52e4f858b52862138bed025 with delete_type TABLET_DATA_DELETED (replaced by tablet 4973e600038742a6bec0bb2d8f3551fe) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.672907 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acde41b2c03440a89243aae008779e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.673111 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 517c09e3bf7046ccba6ed25b08e3460e
I20240506 19:56:37.669659 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 (DEFAULT_TABLE table=default.Table_58c604462fcb4b9cab62687d64877b02 [id=923944e6f77349c3b09903ba96a887fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.673815 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796beb8e0a2c41d698bd4c05796263f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.673974 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.674391 12847 raft_consensus.cc:3037] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.678709 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 2c96f014293242be9640515697c0733b (DEFAULT_TABLE table=default.Table_7f4701bc5da447f58c2c1dad63a0df55 [id=f67e8f35a154427f95ae0f771099e941]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.679680 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c96f014293242be9640515697c0733b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.682565 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5a100640805e4e989b84032702a4de27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.683393 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a100640805e4e989b84032702a4de27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.684623 12519 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148] alter complete (version 5)
I20240506 19:56:37.686152 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.686596 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 (DEFAULT_TABLE table=default.Table_58c604462fcb4b9cab62687d64877b02 [id=923944e6f77349c3b09903ba96a887fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.687045 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796beb8e0a2c41d698bd4c05796263f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.688508 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acde41b2c03440a89243aae008779e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.688773 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472c63d8995346d78633154a8f0d4c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.689445 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 5988b89cb52e4f858b52862138bed025 with delete_type TABLET_DATA_DELETED (replaced by tablet 4973e600038742a6bec0bb2d8f3551fe) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.691147 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581aae0889654903933bef3bf1026954" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:37.696168 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_8be586b55bf2436698bfeb2f94fb465d [id=090dfa9462424be1b846e1cb05569156] alter complete (version 1)
I20240506 19:56:37.700275 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 0114edac976844619e438c17bb959010 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.705109 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 (DEFAULT_TABLE table=default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.705631 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9296bb1104b4cbe279eecd3683796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.706549 13433 consensus_queue.cc:1035] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240506 19:56:37.707943 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e882abe9665a4511bd29de22b6256710" } new_table_name: "default.Table_bb8517f5ef13428c9dcfc8c72d0cab0e" modify_external_catalogs: true
I20240506 19:56:37.708237 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bd8cdb92bf4e49ffb08353c5af21f72e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:37.708595 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 (DEFAULT_TABLE table=default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:37.708859 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd8cdb92bf4e49ffb08353c5af21f72e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.709033 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9296bb1104b4cbe279eecd3683796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.709939 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.710093 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_24bc90d17d404ca3884c13c02624e651" } modify_external_catalogs: true
I20240506 19:56:37.710261 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_24bc90d17d404ca3884c13c02624e651" } modify_external_catalogs: true
I20240506 19:56:37.710889 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_162d0c1d88d04a11b0015afaea52e19f" } modify_external_catalogs: true
I20240506 19:56:37.711160 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_162d0c1d88d04a11b0015afaea52e19f" } modify_external_catalogs: true
I20240506 19:56:37.712095 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff5fed57d64244894f4af1f6067e85) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.712575 12961 tablet_service.cc:1467] Processing CreateTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 (DEFAULT_TABLE table=default.Table_6b562516d9b845789bf9d901c015e37c [id=11d0ad6e93ef421182aaac061edb8501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.713047 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e90b93023e40b3bafbddf7bbcc10b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.715687 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bd20aca236004f33954f66b72b4864e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.716302 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd20aca236004f33954f66b72b4864e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.716260 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_92eb5ef0ea1c408c98a0a14387f76b6e" } new_table_name: "default.Table_44e570af6caa4a749207a8c828639acf" modify_external_catalogs: true
I20240506 19:56:37.718815 12848 raft_consensus.cc:3037] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:37.719449 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 with delete_type TABLET_DATA_DELETED (replaced by tablet da41ef03182445f1946584c2bf015139) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.719573 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca6325447fde4d198454d0ac20e928bd" } modify_external_catalogs: true
I20240506 19:56:37.719753 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca6325447fde4d198454d0ac20e928bd" } modify_external_catalogs: true
I20240506 19:56:37.721786 12848 raft_consensus.cc:1261] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:37.721858 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9cd8092716234d9391b0d018843e289f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.722453 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cd8092716234d9391b0d018843e289f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.722748 13433 consensus_queue.cc:1035] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:37.735736 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3268094cbf481698cb373c1bf6dfba (DEFAULT_TABLE table=default.Table_f29103e88f944dbf883f0de41a88c053 [id=ef86771f67d14d9090bcc7722a834478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.736275 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3268094cbf481698cb373c1bf6dfba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.742111 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 99f4861a01404663870ce91978c8af4f (DEFAULT_TABLE table=default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.742111 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 99f4861a01404663870ce91978c8af4f (DEFAULT_TABLE table=default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:37.742542 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db13b58d17c94e0f82052f823915d9af consensus state has no leader
I20240506 19:56:37.742590 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f4861a01404663870ce91978c8af4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.742597 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f4861a01404663870ce91978c8af4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.743078 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bafd3c4706f74e7da36a8f3538760f43
I20240506 19:56:37.745218 13517 raft_consensus.cc:482] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.745538 13517 raft_consensus.cc:504] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.747820 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet bafd3c4706f74e7da36a8f3538760f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:37.748091 12781 leader_election.cc:341] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:37.748391 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.748493 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0114edac976844619e438c17bb959010
I20240506 19:56:37.748728 12822 tablet_service.cc:1467] Processing CreateTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 (DEFAULT_TABLE table=default.Table_6b562516d9b845789bf9d901c015e37c [id=11d0ad6e93ef421182aaac061edb8501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.749215 13517 leader_election.cc:288] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:37.749475 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e90b93023e40b3bafbddf7bbcc10b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.753715 12517 catalog_manager.cc:6554] ReplaceTablet: tablet 0114edac976844619e438c17bb959010 deleted and replaced by tablet 6352d2c630204df3a24e299e167f92d4
I20240506 19:56:37.754213 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0ba2365d00044c32a06f9017a207adf7
I20240506 19:56:37.756770 13572 tablet.cc:1722] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20240506 19:56:37.759130 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2365d00044c32a06f9017a207adf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.761004 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet bafd3c4706f74e7da36a8f3538760f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.763315 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2365d00044c32a06f9017a207adf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.763648 13551 tablet.cc:1722] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20240506 19:56:37.767841 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bd2c81ac92d04ddc93cc52e5c01cbc60
I20240506 19:56:37.769491 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.774245 13547 raft_consensus.cc:482] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.774437 13547 raft_consensus.cc:504] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:37.775143 13058 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:37.785307 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet d05c5ca7ff864fa98f3494032729e978 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.786466 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ba8c2d366b18431e967a2fcdf655e29e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.787346 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba8c2d366b18431e967a2fcdf655e29e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.792897 12783 leader_election.cc:341] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:37.793146 12783 leader_election.cc:302] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:37.793810 13517 raft_consensus.cc:2726] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:37.797161 13246 ts_tablet_manager.cc:1888] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:37.799629 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.803864 13329 leader_election.cc:288] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.806582 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 (DEFAULT_TABLE table=default.Table_58c604462fcb4b9cab62687d64877b02 [id=923944e6f77349c3b09903ba96a887fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.806764 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 0114edac976844619e438c17bb959010 with delete_type TABLET_DATA_DELETED (replaced by tablet 6352d2c630204df3a24e299e167f92d4) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:37.806948 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 89951.7us
I20240506 19:56:37.803222 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3268094cbf481698cb373c1bf6dfba (DEFAULT_TABLE table=default.Table_f29103e88f944dbf883f0de41a88c053 [id=ef86771f67d14d9090bcc7722a834478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.814931 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 1372be71e1d141ed89e58cdb806431a4 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:37.829085 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_278e4f7429df43d9b9fa9ec82b1d91de" } new_table_name: "default.Table_e43a57ef696d4d749ff2f11fe74f999b" modify_external_catalogs: true
I20240506 19:56:37.829602 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796beb8e0a2c41d698bd4c05796263f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.829617 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502b423bd1040539fb1a3ebbece3c1c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:37.830379 13058 leader_election.cc:341] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:37.830603 13547 leader_election.cc:288] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:37.830655 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 5988b89cb52e4f858b52862138bed025 with delete_type TABLET_DATA_DELETED (replaced by tablet 4973e600038742a6bec0bb2d8f3551fe) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.830724 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 2c96f014293242be9640515697c0733b (DEFAULT_TABLE table=default.Table_7f4701bc5da447f58c2c1dad63a0df55 [id=f67e8f35a154427f95ae0f771099e941]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.831363 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c96f014293242be9640515697c0733b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.832764 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f387dcc396c544b0a400bb57e5e50d05" } new_table_name: "default.Table_75a0ded1f5e044068dfb80dd28b1ae9b" modify_external_catalogs: true
I20240506 19:56:37.830678 13245 tablet_bootstrap.cc:654] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:37.834892 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfeac1a2083945f18e1355bddf3975ff" } new_table_name: "default.Table_c5a00a4676e54afab4a6a801cb898192" modify_external_catalogs: true
W20240506 19:56:37.838539 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17da01620b8b41008421b977455e0c9a has no consensus state
I20240506 19:56:37.844753 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1e8cbfa77d85462baa8b75f47a5c76bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:37.844781 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e75fdc7de2104d288d8d6bf0353108f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.845849 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fce272e5b5f6483c9d031dff1104401b consensus state has no leader
W20240506 19:56:37.845849 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd26e6b8115644cabe6532c7be1bbe95 has no consensus state
W20240506 19:56:37.845968 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e75fdc7de2104d288d8d6bf0353108f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.849889 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_75c7fd329faa49bf99581532e6fd75d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:37.850325 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b666605912f04406bd8d879caa8c8b9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.850517 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75c7fd329faa49bf99581532e6fd75d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.850811 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b666605912f04406bd8d879caa8c8b9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.851197 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e8cbfa77d85462baa8b75f47a5c76bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.855255 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_72cbb579f8c5457ab8ef64876959a3a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.855882 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72cbb579f8c5457ab8ef64876959a3a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.876308 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_247909120dec47d3a2d1729c04ed16a6" } modify_external_catalogs: true
I20240506 19:56:37.876504 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_247909120dec47d3a2d1729c04ed16a6" } modify_external_catalogs: true
I20240506 19:56:37.876222 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_37e9aef6398b47059569bab0a7265a94" } modify_external_catalogs: true
I20240506 19:56:37.880548 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_37e9aef6398b47059569bab0a7265a94" } modify_external_catalogs: true
I20240506 19:56:37.884294 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 0114edac976844619e438c17bb959010 with delete_type TABLET_DATA_DELETED (replaced by tablet 6352d2c630204df3a24e299e167f92d4) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:37.884227 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581aae0889654903933bef3bf1026954" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:37.885813 13246 log.cc:1196] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d0e3260f46024386baac64cb8677883c
W20240506 19:56:37.885855 12922 consensus_peers.cc:485] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a80834187f874661b611b43065eee943. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:37.885980 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 63165.9us
W20240506 19:56:37.886751 13057 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:37.886963 13246 ts_tablet_manager.cc:1909] T d0e3260f46024386baac64cb8677883c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:37.886929 13057 leader_election.cc:302] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:37.887544 13547 raft_consensus.cc:2726] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:37.888712 13546 tablet.cc:1722] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20240506 19:56:37.892850 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502b423bd1040539fb1a3ebbece3c1c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:37.895120 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:37.895382 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_44e570af6caa4a749207a8c828639acf [id=ee08d3829f304e16b5b8994259bb11b0]). Expected version 7 got 6
I20240506 19:56:37.900271 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 00de09ddcdd948a6a3500a6c9d814c14
I20240506 19:56:37.902370 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1372be71e1d141ed89e58cdb806431a4
I20240506 19:56:37.903261 12514 catalog_manager.cc:6554] ReplaceTablet: tablet 1372be71e1d141ed89e58cdb806431a4 deleted and replaced by tablet a1160ab972124630a9be7abeefb4aa56
I20240506 19:56:37.904599 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:37.907048 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3268094cbf481698cb373c1bf6dfba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.908864 13242 ts_tablet_manager.cc:1888] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:37.915985 13242 log.cc:1196] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/03e896b3ba7c4864a26cdfce76fadcfb
I20240506 19:56:37.915788 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_83626d3a5f3348679ec14bda1af29deb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.916496 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83626d3a5f3348679ec14bda1af29deb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.920286 13517 raft_consensus.cc:482] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.920809 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]). Expected version 5 got 0
I20240506 19:56:37.921283 12826 tablet_service.cc:1467] Processing CreateTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 (DEFAULT_TABLE table=default.Table_c5a00a4676e54afab4a6a801cb898192 [id=16258fb4a39c4effb6effa8e857b0b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.922672 13244 ts_tablet_manager.cc:1398] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.349s user 0.005s sys 0.010s
I20240506 19:56:37.921875 12819 tablet_service.cc:1467] Processing CreateTablet for tablet dcb357a7eb2147a79cc69d024bd35269 (DEFAULT_TABLE table=default.Table_bd8cdb92bf4e49ffb08353c5af21f72e [id=b6aeaf0e19d746dab4a37005dc753d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.924930 12823 tablet_service.cc:1467] Processing CreateTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b (DEFAULT_TABLE table=default.Table_6b43eacefdd54cc98fa204f64ee70d7e [id=fb03660d814a4c62b27f9637313a7869]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.926378 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.928551 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3268094cbf481698cb373c1bf6dfba (DEFAULT_TABLE table=default.Table_f29103e88f944dbf883f0de41a88c053 [id=ef86771f67d14d9090bcc7722a834478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.929553 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet bafd3c4706f74e7da36a8f3538760f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.930249 13249 tablet_bootstrap.cc:492] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:37.929313 13517 raft_consensus.cc:504] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.933317 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acde41b2c03440a89243aae008779e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:37.934590 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 4630abf54f024da5be5b45efccc0051d (DEFAULT_TABLE table=default.Table_cae4c030228d49ff88f3c05320bd8924 [id=8aff1566821e4751aa6b062b5959f758]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.932933 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc (DEFAULT_TABLE table=default.Table_ba8c2d366b18431e967a2fcdf655e29e [id=6b539c4e5b2444daa0f9ae05ed98ff54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.936103 13249 ts_tablet_manager.cc:1367] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.357s user 0.073s sys 0.078s
I20240506 19:56:37.937094 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3268094cbf481698cb373c1bf6dfba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.937719 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd26e6b8115644cabe6532c7be1bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.940050 13628 raft_consensus.cc:482] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.940446 13628 raft_consensus.cc:504] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.941804 13103 tablet_service.cc:1467] Processing CreateTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 (DEFAULT_TABLE table=default.Table_6b562516d9b845789bf9d901c015e37c [id=11d0ad6e93ef421182aaac061edb8501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.942274 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e90b93023e40b3bafbddf7bbcc10b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.945057 13517 leader_election.cc:288] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:37.945021 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_788bbfde3ac54d13b047fa3c065680d0" } new_table_name: "default.Table_05d52f607ff842c583d45eafbdcbf2ae" modify_external_catalogs: true
I20240506 19:56:37.945353 13244 tablet_bootstrap.cc:492] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:37.945698 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 1372be71e1d141ed89e58cdb806431a4 with delete_type TABLET_DATA_DELETED (replaced by tablet a1160ab972124630a9be7abeefb4aa56) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.945941 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4630abf54f024da5be5b45efccc0051d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.946496 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df3ab9dff5148f9bb335a7209b29bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.946786 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb357a7eb2147a79cc69d024bd35269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.947993 13242 ts_tablet_manager.cc:1909] T 03e896b3ba7c4864a26cdfce76fadcfb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:37.948503 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9e78a3376c14485f9a70307f926edaa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.949111 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e78a3376c14485f9a70307f926edaa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:37.950178 13242 tablet_replica.cc:330] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:37.950439 13242 raft_consensus.cc:2229] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:37.950896 13242 raft_consensus.cc:2258] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:37.951700 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0f74c7c4cd42d293fa244de3fd693b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.952255 13517 raft_consensus.cc:482] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:37.952306 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_56eb06f51079472894642f592bbe43a3" } new_table_name: "default.Table_5a18ae8689874543a57ab917b1efaff0" modify_external_catalogs: true
I20240506 19:56:37.952462 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc" } modify_external_catalogs: true
I20240506 19:56:37.952632 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc" } modify_external_catalogs: true
I20240506 19:56:37.953248 13517 raft_consensus.cc:504] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:37.954756 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 (DEFAULT_TABLE table=default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.954823 13517 leader_election.cc:288] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:37.955204 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9296bb1104b4cbe279eecd3683796. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:37.956398 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:37.958719 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:37.962671 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 304a1401627948bbaff78fae4d42ab22 consensus state has no leader
I20240506 19:56:37.964670 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e87268955e8247709b420f45ae2e4e5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:37.964843 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5217c257ff46e48c23563e496b6f08 (DEFAULT_TABLE table=default.Table_46d49a82b855486f994526543440ace7 [id=4013fcd5ff0545f28ba8f7f5cfc64345]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:37.965307 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e87268955e8247709b420f45ae2e4e5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:37.965519 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a6e0126d9854215a0e174a76c0a0391 consensus state has no leader
I20240506 19:56:37.966297 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cc8e3aa90fea4998b3e0a4a411add544" } modify_external_catalogs: true
I20240506 19:56:37.966522 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cc8e3aa90fea4998b3e0a4a411add544" } modify_external_catalogs: true
I20240506 19:56:37.967545 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c (DEFAULT_TABLE table=default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.968091 12816 tablet_service.cc:1467] Processing CreateTablet for tablet b2be78cbc8264f6e8d2e05137135161c (DEFAULT_TABLE table=default.Table_bd20aca236004f33954f66b72b4864e6 [id=93500e72118e41ffac81ba66586212d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.968952 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 (DEFAULT_TABLE table=default.Table_6d22301588d54dcd8649bc08a1ade8f6 [id=09c0d79ce0d443079ca27266e1154dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.969938 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 6352d2c630204df3a24e299e167f92d4 (DEFAULT_TABLE table=default.Table_1ba0602f38e64789a126e75fc115f5c2 [id=c2803598df8241beb0b6ce684515661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.970155 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_54f3105144e741119c67aec80c408fa5" } modify_external_catalogs: true
I20240506 19:56:37.970346 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_54f3105144e741119c67aec80c408fa5" } modify_external_catalogs: true
I20240506 19:56:37.970613 12820 tablet_service.cc:1467] Processing CreateTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b (DEFAULT_TABLE table=default.Table_54f3105144e741119c67aec80c408fa5 [id=6369da651bef4fa58ae1c1711da138b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.971561 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 04738def56324a368f8182543107e32e (DEFAULT_TABLE table=default.Table_0c42cb0cd7e349fa86150517492387bb [id=52a0a57c5c19479e90684d073873755b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.975992 12813 tablet_service.cc:1467] Processing CreateTablet for tablet f3e748d2c32a4b09abc872fed318e643 (DEFAULT_TABLE table=default.Table_9cd8092716234d9391b0d018843e289f [id=0f7c418d978f4813bf5db9a9e31bc24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:37.987617 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_211f1d00bb6e407b90eac9c696f99e83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:37.988315 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_211f1d00bb6e407b90eac9c696f99e83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.000016 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e737c1fa78974f05a022649a900e2636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.000708 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e737c1fa78974f05a022649a900e2636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.012094 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f3e7329b22414da886dee4e1ec6bf3ad
I20240506 19:56:38.014355 13124 raft_consensus.cc:1261] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:38.014935 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 255) took 1115 ms. Trace:
I20240506 19:56:38.015072 13246 rpcz_store.cc:276] 0506 19:56:36.899384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.899746 (+ 362us) service_pool.cc:225] Handling call
0506 19:56:37.249105 (+349359us) ts_tablet_manager.cc:1120] Deleting tablet d0e3260f46024386baac64cb8677883c
0506 19:56:37.249115 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.519309 (+270194us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:37.797014 (+277705us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.014870 (+217856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":349154}
I20240506 19:56:38.015556 13246 tablet_replica.cc:330] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.015766 13246 raft_consensus.cc:2229] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:38.016052 13246 pending_rounds.cc:62] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:38.016127 13246 pending_rounds.cc:68] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744026918252544 op_type: NO_OP noop_request { }
I20240506 19:56:38.016285 13246 raft_consensus.cc:2866] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:38.016398 13246 raft_consensus.cc:2258] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.019264 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 0114edac976844619e438c17bb959010 with delete_type TABLET_DATA_DELETED (replaced by tablet 6352d2c630204df3a24e299e167f92d4) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.021157 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2365d00044c32a06f9017a207adf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.022835 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.027287 13103 tablet_service.cc:1467] Processing CreateTablet for tablet dcb357a7eb2147a79cc69d024bd35269 (DEFAULT_TABLE table=default.Table_bd8cdb92bf4e49ffb08353c5af21f72e [id=b6aeaf0e19d746dab4a37005dc753d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.028149 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb357a7eb2147a79cc69d024bd35269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.031392 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 99f4861a01404663870ce91978c8af4f (DEFAULT_TABLE table=default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.031984 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f4861a01404663870ce91978c8af4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.032481 13095 tablet_service.cc:1467] Processing CreateTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 (DEFAULT_TABLE table=default.Table_c5a00a4676e54afab4a6a801cb898192 [id=16258fb4a39c4effb6effa8e857b0b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.033026 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd26e6b8115644cabe6532c7be1bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.036209 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc (DEFAULT_TABLE table=default.Table_ba8c2d366b18431e967a2fcdf655e29e [id=6b539c4e5b2444daa0f9ae05ed98ff54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.036695 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df3ab9dff5148f9bb335a7209b29bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.037421 13433 raft_consensus.cc:482] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.038432 13433 raft_consensus.cc:504] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.039229 13090 tablet_service.cc:1467] Processing CreateTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b (DEFAULT_TABLE table=default.Table_6b43eacefdd54cc98fa204f64ee70d7e [id=fb03660d814a4c62b27f9637313a7869]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.039624 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0f74c7c4cd42d293fa244de3fd693b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.040429 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_915828e8f55f4e1687a45a7fe7fa4f34" } new_table_name: "default.Table_d9668767a7184407ac5f03e85864411e" modify_external_catalogs: true
I20240506 19:56:38.041270 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet 1372be71e1d141ed89e58cdb806431a4 with delete_type TABLET_DATA_DELETED (replaced by tablet a1160ab972124630a9be7abeefb4aa56) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.041743 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 4630abf54f024da5be5b45efccc0051d (DEFAULT_TABLE table=default.Table_cae4c030228d49ff88f3c05320bd8924 [id=8aff1566821e4751aa6b062b5959f758]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.042186 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4630abf54f024da5be5b45efccc0051d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.042349 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff5fed57d64244894f4af1f6067e85) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.043361 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.044698 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5217c257ff46e48c23563e496b6f08 (DEFAULT_TABLE table=default.Table_46d49a82b855486f994526543440ace7 [id=4013fcd5ff0545f28ba8f7f5cfc64345]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.045056 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5217c257ff46e48c23563e496b6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.045430 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.045675 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7f56fea34946e590dae0621b28f9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d23f0a5ff6d430282256e9bddc186b9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.045908 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 581aae0889654903933bef3bf1026954 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f56d2180b1472a8168fb3a63589065) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:38.054250 12506 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1b7f56fea34946e590dae0621b28f9c6 (table default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b7f56fea34946e590dae0621b28f9c6 consensus state has no leader
I20240506 19:56:38.054776 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1e4ddadc6b5a4ce9810916112217bff2" } new_table_name: "default.Table_a495d810a1ce4d3a80b146059120ad17" modify_external_catalogs: true
I20240506 19:56:38.055230 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c3908be635fe46a6a6efe79f7916080d" } new_table_name: "default.Table_4da2e480af98486fa1887da89cc86e38" modify_external_catalogs: true
I20240506 19:56:38.053681 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c (DEFAULT_TABLE table=default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.056308 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7c61244620487bbbac26dcdbf8a38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.061655 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c0d358df6dd84d60a343a335303508eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.062361 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0d358df6dd84d60a343a335303508eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.069145 13098 tablet_service.cc:1467] Processing CreateTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b (DEFAULT_TABLE table=default.Table_54f3105144e741119c67aec80c408fa5 [id=6369da651bef4fa58ae1c1711da138b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.070137 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeae70e0d1144ea9e3e0f8727c15e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.071228 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5516e8aad4e8474692b5b6c40bc483c1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:38.071555 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e3f798d6484d4fabd55f9b4c98a911" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:38.076948 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 04738def56324a368f8182543107e32e (DEFAULT_TABLE table=default.Table_0c42cb0cd7e349fa86150517492387bb [id=52a0a57c5c19479e90684d073873755b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.080755 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04738def56324a368f8182543107e32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.081935 13095 tablet_service.cc:1467] Processing CreateTablet for tablet b2be78cbc8264f6e8d2e05137135161c (DEFAULT_TABLE table=default.Table_bd20aca236004f33954f66b72b4864e6 [id=93500e72118e41ffac81ba66586212d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.082533 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2be78cbc8264f6e8d2e05137135161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.084439 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c with delete_type TABLET_DATA_DELETED (replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.084473 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 (DEFAULT_TABLE table=default.Table_6d22301588d54dcd8649bc08a1ade8f6 [id=09c0d79ce0d443079ca27266e1154dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.085141 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff3435c11854649b7daf74f3d8c33d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.085390 13090 tablet_service.cc:1467] Processing CreateTablet for tablet f3e748d2c32a4b09abc872fed318e643 (DEFAULT_TABLE table=default.Table_9cd8092716234d9391b0d018843e289f [id=0f7c418d978f4813bf5db9a9e31bc24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.085747 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e748d2c32a4b09abc872fed318e643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.086633 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 (DEFAULT_TABLE table=default.Table_4cbe3de6b961432fbeacafa9ae5fe50c [id=f6e95966d94c4b5694b843e272ea941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.087019 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234e4f3439e84b5fbb2f7bab1dbd0822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.087921 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 17da01620b8b41008421b977455e0c9a (DEFAULT_TABLE table=default.Table_e43a57ef696d4d749ff2f11fe74f999b [id=a445760bb4514251b6ff954ca946cd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.088255 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17da01620b8b41008421b977455e0c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.089411 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 6352d2c630204df3a24e299e167f92d4 (DEFAULT_TABLE table=default.Table_1ba0602f38e64789a126e75fc115f5c2 [id=c2803598df8241beb0b6ce684515661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.090106 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6352d2c630204df3a24e299e167f92d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.088907 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 (DEFAULT_TABLE table=default.Table_4cbe3de6b961432fbeacafa9ae5fe50c [id=f6e95966d94c4b5694b843e272ea941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.090768 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 4973e600038742a6bec0bb2d8f3551fe (DEFAULT_TABLE table=default.Table_ac80e9e0789c4cbaab8826eb3c0bc621 [id=cd2382012b444dd09cd9ad1a50a20b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.091265 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 03e896b3ba7c4864a26cdfce76fadcfb (table default.Table_fe77d7d403a041b4b9f6a1c97177f461 [id=15e39c71182c4a089e9c950679153305]) successfully deleted
I20240506 19:56:38.091460 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4973e600038742a6bec0bb2d8f3551fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.091898 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a8b3ed0b56aa478abb44347afca8fcf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8b3ed0b56aa478abb44347afca8fcf6 already in progress: creating tablet
I20240506 19:56:38.092018 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5217c257ff46e48c23563e496b6f08. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.092540 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 472c63d8995346d78633154a8f0d4c2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472c63d8995346d78633154a8f0d4c2b already in progress: creating tablet
I20240506 19:56:38.092818 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:38.093712 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2f7f7171d06948388bc0fdb3514202ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7f7171d06948388bc0fdb3514202ed already in progress: creating tablet
W20240506 19:56:38.094115 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b899fb38bfed4e02b3708f6abec80e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b899fb38bfed4e02b3708f6abec80e0a already in progress: creating tablet
W20240506 19:56:38.094568 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d05c5ca7ff864fa98f3494032729e978 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d05c5ca7ff864fa98f3494032729e978 already in progress: creating tablet
I20240506 19:56:38.089983 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 4973e600038742a6bec0bb2d8f3551fe (DEFAULT_TABLE table=default.Table_ac80e9e0789c4cbaab8826eb3c0bc621 [id=cd2382012b444dd09cd9ad1a50a20b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.089860 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 17da01620b8b41008421b977455e0c9a (DEFAULT_TABLE table=default.Table_e43a57ef696d4d749ff2f11fe74f999b [id=a445760bb4514251b6ff954ca946cd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.118769 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 (DEFAULT_TABLE table=default.Table_d909ad6f09e24952918c9471609c9d27 [id=2ebd467d8ac3476192b5634ac512aa47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.126140 13246 ts_tablet_manager.cc:1875] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.129647 13433 leader_election.cc:288] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:38.135385 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:38.136240 12783 leader_election.cc:341] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d843fe82af1148f4a9045ce8b09ad11d
I20240506 19:56:38.136955 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.137058 12809 tablet_service.cc:1467] Processing CreateTablet for tablet f8ce7f4ceb9d40009e9c75543b1fc545 (DEFAULT_TABLE table=default.Table_5a100640805e4e989b84032702a4de27 [id=7d2da7abfa3e48f79147c8a16820ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.140820 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 (DEFAULT_TABLE table=default.Table_4cbe3de6b961432fbeacafa9ae5fe50c [id=f6e95966d94c4b5694b843e272ea941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.141489 12958 tablet_service.cc:1467] Processing CreateTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 (DEFAULT_TABLE table=default.Table_c5a00a4676e54afab4a6a801cb898192 [id=16258fb4a39c4effb6effa8e857b0b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.142802 12954 tablet_service.cc:1467] Processing CreateTablet for tablet f3e748d2c32a4b09abc872fed318e643 (DEFAULT_TABLE table=default.Table_9cd8092716234d9391b0d018843e289f [id=0f7c418d978f4813bf5db9a9e31bc24d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:38.142975 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 215476us
I20240506 19:56:38.143309 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 1372be71e1d141ed89e58cdb806431a4 with delete_type TABLET_DATA_DELETED (replaced by tablet a1160ab972124630a9be7abeefb4aa56) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:38.143417 13057 leader_election.cc:341] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.143589 13057 leader_election.cc:302] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.144065 13625 raft_consensus.cc:2726] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.145270 13517 raft_consensus.cc:482] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.145572 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e748d2c32a4b09abc872fed318e643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.145287 13572 tablet.cc:1722] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240506 19:56:38.145931 12921 leader_election.cc:341] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:38.146164 12921 leader_election.cc:341] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.146538 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5516e8aad4e8474692b5b6c40bc483c1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:38.146622 12955 tablet_service.cc:1467] Processing CreateTablet for tablet dcb357a7eb2147a79cc69d024bd35269 (DEFAULT_TABLE table=default.Table_bd8cdb92bf4e49ffb08353c5af21f72e [id=b6aeaf0e19d746dab4a37005dc753d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.147344 13546 tablet.cc:1722] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240506 19:56:38.147538 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e3f798d6484d4fabd55f9b4c98a911" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:38.148159 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 4630abf54f024da5be5b45efccc0051d (DEFAULT_TABLE table=default.Table_cae4c030228d49ff88f3c05320bd8924 [id=8aff1566821e4751aa6b062b5959f758]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.148088 13249 raft_consensus.cc:348] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.148380 13249 raft_consensus.cc:374] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.148381 13124 raft_consensus.cc:2454] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:38.148478 13249 raft_consensus.cc:729] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.148094 12959 tablet_service.cc:1467] Processing CreateTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b (DEFAULT_TABLE table=default.Table_54f3105144e741119c67aec80c408fa5 [id=6369da651bef4fa58ae1c1711da138b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.148741 13249 consensus_queue.cc:260] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.150353 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c (DEFAULT_TABLE table=default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.150708 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 04738def56324a368f8182543107e32e (DEFAULT_TABLE table=default.Table_0c42cb0cd7e349fa86150517492387bb [id=52a0a57c5c19479e90684d073873755b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.151026 12963 tablet_service.cc:1467] Processing CreateTablet for tablet b2be78cbc8264f6e8d2e05137135161c (DEFAULT_TABLE table=default.Table_bd20aca236004f33954f66b72b4864e6 [id=93500e72118e41ffac81ba66586212d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.151330 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc (DEFAULT_TABLE table=default.Table_ba8c2d366b18431e967a2fcdf655e29e [id=6b539c4e5b2444daa0f9ae05ed98ff54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.149624 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 (DEFAULT_TABLE table=default.Table_d909ad6f09e24952918c9471609c9d27 [id=2ebd467d8ac3476192b5634ac512aa47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.158803 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]). Expected version 1 got 0
I20240506 19:56:38.159755 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_27b96f0770c845b38361dfc9ac3f86f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.160396 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27b96f0770c845b38361dfc9ac3f86f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.162281 13245 tablet_bootstrap.cc:492] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:38.162942 13099 tablet_service.cc:1467] Processing CreateTablet for tablet f8ce7f4ceb9d40009e9c75543b1fc545 (DEFAULT_TABLE table=default.Table_5a100640805e4e989b84032702a4de27 [id=7d2da7abfa3e48f79147c8a16820ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.163861 13245 ts_tablet_manager.cc:1367] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.607s user 0.023s sys 0.027s
I20240506 19:56:38.169229 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bc7cfd006f4d4ca95541dab6873d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.170141 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ce7f4ceb9d40009e9c75543b1fc545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.171072 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.171769 13245 raft_consensus.cc:348] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.173552 13245 raft_consensus.cc:374] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.173660 13245 raft_consensus.cc:729] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.177577 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4973e600038742a6bec0bb2d8f3551fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.179841 13517 raft_consensus.cc:504] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:38.180128 13058 consensus_peers.cc:485] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03e896b3ba7c4864a26cdfce76fadcfb. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:38.181807 13249 ts_tablet_manager.cc:1398] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.244s user 0.003s sys 0.000s
I20240506 19:56:38.182343 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d0e3260f46024386baac64cb8677883c (table default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]) successfully deleted
I20240506 19:56:38.174582 13245 consensus_queue.cc:260] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.186925 13245 ts_tablet_manager.cc:1398] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240506 19:56:38.187340 13245 tablet_bootstrap.cc:492] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:38.189121 13245 tablet_bootstrap.cc:654] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:38.190920 13242 ts_tablet_manager.cc:1875] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.209491 13246 ts_tablet_manager.cc:1888] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:38.210101 13246 log.cc:1196] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:38.211854 13246 ts_tablet_manager.cc:1909] T a16704567ef6454eaef886fb9cd599ce P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:38.215409 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 consensus state has no leader
I20240506 19:56:38.215754 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b40bb86f59744dc387a12a83d5dc8b9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:38.216154 12519 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_44e570af6caa4a749207a8c828639acf [id=ee08d3829f304e16b5b8994259bb11b0] alter complete (version 7)
W20240506 19:56:38.216533 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b40bb86f59744dc387a12a83d5dc8b9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.216408 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 253) took 1317 ms. Trace:
I20240506 19:56:38.216871 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b8e4c400be8f434295e3730c95d14641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:38.216732 13246 rpcz_store.cc:276] 0506 19:56:36.899116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.899990 (+ 874us) service_pool.cc:225] Handling call
0506 19:56:38.015475 (+1115485us) ts_tablet_manager.cc:1120] Deleting tablet a16704567ef6454eaef886fb9cd599ce
0506 19:56:38.015486 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.195997 (+180511us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.208960 (+ 12963us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.216137 (+ 7177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115273}
W20240506 19:56:38.217482 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8e4c400be8f434295e3730c95d14641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:38.218657 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e3d13322b4a4a2db3b6d425679f68ed consensus state has no leader
I20240506 19:56:38.218623 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 256) took 1173 ms. Trace:
I20240506 19:56:38.219033 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e43a57ef696d4d749ff2f11fe74f999b" } modify_external_catalogs: true
I20240506 19:56:38.215799 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2e36b2f9be7f4af9a58215847e671f2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:38.219214 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e43a57ef696d4d749ff2f11fe74f999b" } modify_external_catalogs: true
W20240506 19:56:38.219527 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e36b2f9be7f4af9a58215847e671f2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.219841 13599 log.cc:824] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:38.219010 13246 rpcz_store.cc:276] 0506 19:56:37.044736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.044809 (+ 73us) service_pool.cc:225] Handling call
0506 19:56:38.217944 (+1173135us) ts_tablet_manager.cc:1120] Deleting tablet da41ef03182445f1946584c2bf015139
0506 19:56:38.218003 (+ 59us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.218462 (+ 459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172804}
I20240506 19:56:38.225720 13246 tablet_replica.cc:330] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.227742 13246 raft_consensus.cc:2229] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:38.228607 13246 pending_rounds.cc:62] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:38.228873 13246 pending_rounds.cc:68] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744026443476992 op_type: NO_OP noop_request { }
I20240506 19:56:38.232168 13246 raft_consensus.cc:2866] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:38.232705 13246 raft_consensus.cc:2258] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.235294 13246 ts_tablet_manager.cc:1875] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.237920 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 6352d2c630204df3a24e299e167f92d4 (DEFAULT_TABLE table=default.Table_1ba0602f38e64789a126e75fc115f5c2 [id=c2803598df8241beb0b6ce684515661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.238611 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 (DEFAULT_TABLE table=default.Table_6d22301588d54dcd8649bc08a1ade8f6 [id=09c0d79ce0d443079ca27266e1154dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.242632 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234e4f3439e84b5fbb2f7bab1dbd0822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.242884 13245 tablet_bootstrap.cc:492] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:38.243129 13245 ts_tablet_manager.cc:1367] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.013s
I20240506 19:56:38.243219 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeae70e0d1144ea9e3e0f8727c15e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.243444 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e748d2c32a4b09abc872fed318e643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.243985 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4630abf54f024da5be5b45efccc0051d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.244580 13245 raft_consensus.cc:348] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.244928 13245 raft_consensus.cc:374] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.245126 13245 raft_consensus.cc:729] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.245621 13245 consensus_queue.cc:260] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.247443 13245 ts_tablet_manager.cc:1398] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:38.248569 13245 tablet_bootstrap.cc:492] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:38.249346 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5217c257ff46e48c23563e496b6f08 (DEFAULT_TABLE table=default.Table_46d49a82b855486f994526543440ace7 [id=4013fcd5ff0545f28ba8f7f5cfc64345]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.249861 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5217c257ff46e48c23563e496b6f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.251987 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff3435c11854649b7daf74f3d8c33d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.253968 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 (DEFAULT_TABLE table=default.Table_d909ad6f09e24952918c9471609c9d27 [id=2ebd467d8ac3476192b5634ac512aa47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.254402 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bc7cfd006f4d4ca95541dab6873d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.255313 13245 tablet_bootstrap.cc:654] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:38.255416 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6352d2c630204df3a24e299e167f92d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.261174 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd26e6b8115644cabe6532c7be1bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.262027 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df3ab9dff5148f9bb335a7209b29bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.264065 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 17da01620b8b41008421b977455e0c9a (DEFAULT_TABLE table=default.Table_e43a57ef696d4d749ff2f11fe74f999b [id=a445760bb4514251b6ff954ca946cd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.264626 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17da01620b8b41008421b977455e0c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.265563 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04738def56324a368f8182543107e32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.269444 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7c61244620487bbbac26dcdbf8a38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.273676 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2be78cbc8264f6e8d2e05137135161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.276080 13246 ts_tablet_manager.cc:1888] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:38.276368 13246 log.cc:1196] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/92951843c8bd446aa86436b65d9eb793
I20240506 19:56:38.277350 13246 ts_tablet_manager.cc:1909] T 92951843c8bd446aa86436b65d9eb793 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:38.277587 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb357a7eb2147a79cc69d024bd35269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.282158 13249 tablet_bootstrap.cc:492] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:38.283432 13246 tablet_replica.cc:330] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.284488 13249 tablet_bootstrap.cc:654] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:38.284344 13246 raft_consensus.cc:2229] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:38.285683 13246 raft_consensus.cc:2258] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.288442 13246 ts_tablet_manager.cc:1875] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.290784 12847 raft_consensus.cc:3037] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:38.291226 13551 tablet.cc:1722] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:38.299660 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:38.300019 12988 raft_consensus.cc:2379] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef18e88b87064ed4b6c20f9325ebcc33 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:38.302477 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234e4f3439e84b5fbb2f7bab1dbd0822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.308706 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a16704567ef6454eaef886fb9cd599ce (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:38.309092 13246 ts_tablet_manager.cc:1888] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:38.309398 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bc7cfd006f4d4ca95541dab6873d32. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.309710 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a8b3ed0b56aa478abb44347afca8fcf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8b3ed0b56aa478abb44347afca8fcf6 already in progress: creating tablet
W20240506 19:56:38.310002 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7d23f0a5ff6d430282256e9bddc186b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d23f0a5ff6d430282256e9bddc186b9 already in progress: creating tablet
W20240506 19:56:38.310184 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4a080ccac6394210b0b24b65843f9d75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a080ccac6394210b0b24b65843f9d75 already in progress: creating tablet
I20240506 19:56:38.310650 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 92951843c8bd446aa86436b65d9eb793 (table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]) successfully deleted
W20240506 19:56:38.310848 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 64b7e42061cd487b8517447eb59ea8a1 because the tablet was not found. No further retry: Not found: Tablet not found: 64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:38.310927 13246 log.cc:1196] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/03e896b3ba7c4864a26cdfce76fadcfb
I20240506 19:56:38.311411 13246 ts_tablet_manager.cc:1909] T 03e896b3ba7c4864a26cdfce76fadcfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:38.312496 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 03e896b3ba7c4864a26cdfce76fadcfb (table default.Table_fe77d7d403a041b4b9f6a1c97177f461 [id=15e39c71182c4a089e9c950679153305]) successfully deleted
W20240506 19:56:38.314503 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2f7f7171d06948388bc0fdb3514202ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7f7171d06948388bc0fdb3514202ed already in progress: creating tablet
I20240506 19:56:38.315233 13632 consensus_queue.cc:1035] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.191s
I20240506 19:56:38.315877 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=17 ignored=0} inserts{seen=23 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240506 19:56:38.315936 12960 tablet_service.cc:1467] Processing CreateTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b (DEFAULT_TABLE table=default.Table_6b43eacefdd54cc98fa204f64ee70d7e [id=fb03660d814a4c62b27f9637313a7869]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:38.316114 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b899fb38bfed4e02b3708f6abec80e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b899fb38bfed4e02b3708f6abec80e0a already in progress: creating tablet
I20240506 19:56:38.316323 12949 tablet_service.cc:1467] Processing CreateTablet for tablet f8ce7f4ceb9d40009e9c75543b1fc545 (DEFAULT_TABLE table=default.Table_5a100640805e4e989b84032702a4de27 [id=7d2da7abfa3e48f79147c8a16820ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.316323 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0f74c7c4cd42d293fa244de3fd693b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.316700 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: Bootstrap complete.
W20240506 19:56:38.316918 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 472c63d8995346d78633154a8f0d4c2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472c63d8995346d78633154a8f0d4c2b already in progress: creating tablet
W20240506 19:56:38.317785 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d05c5ca7ff864fa98f3494032729e978 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d05c5ca7ff864fa98f3494032729e978 already in progress: creating tablet
W20240506 19:56:38.318598 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5988b89cb52e4f858b52862138bed025 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5988b89cb52e4f858b52862138bed025 already in progress: creating tablet
W20240506 19:56:38.319562 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 517c09e3bf7046ccba6ed25b08e3460e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 517c09e3bf7046ccba6ed25b08e3460e already in progress: creating tablet
W20240506 19:56:38.320294 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bafd3c4706f74e7da36a8f3538760f43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bafd3c4706f74e7da36a8f3538760f43 already in progress: creating tablet
I20240506 19:56:38.320420 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ce7f4ceb9d40009e9c75543b1fc545. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.321048 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4acde41b2c03440a89243aae008779e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4acde41b2c03440a89243aae008779e1 already in progress: creating tablet
I20240506 19:56:38.315843 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 4973e600038742a6bec0bb2d8f3551fe (DEFAULT_TABLE table=default.Table_ac80e9e0789c4cbaab8826eb3c0bc621 [id=cd2382012b444dd09cd9ad1a50a20b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.321730 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4973e600038742a6bec0bb2d8f3551fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.321835 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0114edac976844619e438c17bb959010 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0114edac976844619e438c17bb959010 already in progress: creating tablet
W20240506 19:56:38.322597 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0ba2365d00044c32a06f9017a207adf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ba2365d00044c32a06f9017a207adf7 already in progress: creating tablet
W20240506 19:56:38.323402 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd2c81ac92d04ddc93cc52e5c01cbc60 already in progress: creating tablet
W20240506 19:56:38.324204 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1372be71e1d141ed89e58cdb806431a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1372be71e1d141ed89e58cdb806431a4 already in progress: creating tablet
I20240506 19:56:38.324685 13246 tablet_replica.cc:330] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.324914 13246 raft_consensus.cc:2229] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:38.325554 13246 raft_consensus.cc:2258] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:38.326400 12783 leader_election.cc:341] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 30cb3cb7cbe14cc9bf1bfd07227f538e
I20240506 19:56:38.327445 13246 ts_tablet_manager.cc:1875] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.339244 13632 consensus_queue.cc:1035] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:38.340126 12922 leader_election.cc:341] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:38.342669 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1f1e4efbe8b40b199fcb3020569e1a1 consensus state has no leader
I20240506 19:56:38.347561 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:56:38.347645 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 with delete_type TABLET_DATA_DELETED (replaced by tablet da41ef03182445f1946584c2bf015139) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.348574 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3a6afcad38024709a1f104fa5bbadcff" } modify_external_catalogs: true
I20240506 19:56:38.348798 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3a6afcad38024709a1f104fa5bbadcff" } modify_external_catalogs: true
I20240506 19:56:38.350255 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.351255 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:38.356580 13599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:38.365646 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8bb7368b19d441baa65305aa9e64514d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.366487 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bb7368b19d441baa65305aa9e64514d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.373365 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.374155 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.376675 13246 ts_tablet_manager.cc:1888] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:38.376845 13246 log.cc:1196] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/c5e3f798d6484d4fabd55f9b4c98a911
I20240506 19:56:38.377259 13246 ts_tablet_manager.cc:1909] T c5e3f798d6484d4fabd55f9b4c98a911 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:38.380790 13599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d4482cd59a4e5687992b1f8a177373, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.381515 13246 tablet_replica.cc:330] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.381724 13246 raft_consensus.cc:2229] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:38.381829 13599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:38.382467 13246 raft_consensus.cc:2258] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.383088 13246 ts_tablet_manager.cc:1875] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.385691 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet c5e3f798d6484d4fabd55f9b4c98a911 (table default.Table_d08a9f7decde4bb69b28102c4b21a5bd [id=249eebb1852745929487a8fff7406ca1]) successfully deleted
W20240506 19:56:38.386019 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 6b59ad067c274fddbba5f71e89480dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b59ad067c274fddbba5f71e89480dec already in progress: creating tablet
W20240506 19:56:38.386281 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 00de09ddcdd948a6a3500a6c9d814c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00de09ddcdd948a6a3500a6c9d814c14 already in progress: creating tablet
I20240506 19:56:38.389923 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:38.391336 13245 tablet_bootstrap.cc:492] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:38.391484 13517 leader_election.cc:288] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.391587 13245 ts_tablet_manager.cc:1367] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.143s user 0.020s sys 0.027s
I20240506 19:56:38.392402 13245 raft_consensus.cc:348] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.392765 13245 raft_consensus.cc:374] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.392861 13245 raft_consensus.cc:729] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.393150 13245 consensus_queue.cc:260] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.395745 13245 ts_tablet_manager.cc:1398] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:38.396131 13245 tablet_bootstrap.cc:492] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:38.399724 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:38.399974 13245 tablet_bootstrap.cc:654] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:38.392702 13599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 19:56:38.408310 13245 tablet_bootstrap.cc:492] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:38.408501 13245 ts_tablet_manager.cc:1367] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240506 19:56:38.409693 13245 raft_consensus.cc:348] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.410040 13245 raft_consensus.cc:374] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.410142 13245 raft_consensus.cc:729] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.410709 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_65407acf1df5470f8ec9e008fdbcbd93" } new_table_name: "default.Table_a2fb551b2fc541e38f0a655c4539060b" modify_external_catalogs: true
I20240506 19:56:38.410811 13245 consensus_queue.cc:260] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.411417 13245 ts_tablet_manager.cc:1398] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 19:56:38.411836 13245 tablet_bootstrap.cc:492] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:38.412310 13171 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:38.413537 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7f4701bc5da447f58c2c1dad63a0df55" } new_table_name: "default.Table_a8f2511323134e30aeb69565f63f961f" modify_external_catalogs: true
I20240506 19:56:38.414232 13245 tablet_bootstrap.cc:654] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:38.422298 12516 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:38.423425 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c3e795b5a9724754a33f79ebe9ac2c1f" } modify_external_catalogs: true
I20240506 19:56:38.423712 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c3e795b5a9724754a33f79ebe9ac2c1f" } modify_external_catalogs: true
I20240506 19:56:38.423993 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:38.428195 12520 catalog_manager.cc:5484] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:38.430689 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 116df9bd6ab04a30ba09861e765d7267 (DEFAULT_TABLE table=default.Table_9e78a3376c14485f9a70307f926edaa5 [id=db8bd7495bce4c518581981bca778450]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.431411 13091 tablet_service.cc:1467] Processing CreateTablet for tablet d9443cdbaa3c40299b66d216673c4b1a (DEFAULT_TABLE table=default.Table_1e8cbfa77d85462baa8b75f47a5c76bf [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.432004 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 08dafb661b2443f5afadc6816868bac8 (DEFAULT_TABLE table=default.Table_72cbb579f8c5457ab8ef64876959a3a5 [id=8c16005dc18b44a4918959ca3868f9f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.432559 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb (DEFAULT_TABLE table=default.Table_e75fdc7de2104d288d8d6bf0353108f3 [id=4884fa87524f4bdebb7c9e61f25a48a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.433073 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 (DEFAULT_TABLE table=default.Table_b666605912f04406bd8d879caa8c8b9b [id=85453d1f205a4fc5bcf2dd05063162f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.433642 13093 tablet_service.cc:1467] Processing CreateTablet for tablet a1160ab972124630a9be7abeefb4aa56 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.434265 13097 tablet_service.cc:1467] Processing CreateTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 (DEFAULT_TABLE table=default.Table_83626d3a5f3348679ec14bda1af29deb [id=034d601564f94db89efafdf7d3b35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.435243 13094 tablet_service.cc:1467] Processing CreateTablet for tablet d7be9788df8e47d285bd932d078c6a8b (DEFAULT_TABLE table=default.Table_75c7fd329faa49bf99581532e6fd75d2 [id=8baacc24dda44dd5ad46aef596e96276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.438686 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 116df9bd6ab04a30ba09861e765d7267 (DEFAULT_TABLE table=default.Table_9e78a3376c14485f9a70307f926edaa5 [id=db8bd7495bce4c518581981bca778450]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.445178 13245 tablet_bootstrap.cc:492] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:38.447212 13245 ts_tablet_manager.cc:1367] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.008s
I20240506 19:56:38.448447 13245 raft_consensus.cc:348] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.448782 13245 raft_consensus.cc:374] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.448880 13245 raft_consensus.cc:729] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.453356 12823 tablet_service.cc:1467] Processing CreateTablet for tablet d9443cdbaa3c40299b66d216673c4b1a (DEFAULT_TABLE table=default.Table_1e8cbfa77d85462baa8b75f47a5c76bf [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.456962 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 116df9bd6ab04a30ba09861e765d7267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.457419 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9443cdbaa3c40299b66d216673c4b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.457618 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06187c83b94d03ae4ebb67a2873cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.457968 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d1f01c0edb4f6b8a621c1643ca1afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.458207 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7be9788df8e47d285bd932d078c6a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.458415 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dafb661b2443f5afadc6816868bac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.461598 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997ecba7fbae41178a25bf7af4d152e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.461850 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1160ab972124630a9be7abeefb4aa56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.458173 13245 consensus_queue.cc:260] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.467386 13245 ts_tablet_manager.cc:1398] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240506 19:56:38.467918 13245 tablet_bootstrap.cc:492] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:38.472254 13245 tablet_bootstrap.cc:654] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:38.473416 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a080ccac6394210b0b24b65843f9d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.474296 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8a9ecf47d7474c82758417449d940c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.477670 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 (DEFAULT_TABLE table=default.Table_b666605912f04406bd8d879caa8c8b9b [id=85453d1f205a4fc5bcf2dd05063162f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.478181 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb (DEFAULT_TABLE table=default.Table_e75fdc7de2104d288d8d6bf0353108f3 [id=4884fa87524f4bdebb7c9e61f25a48a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.478931 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.479780 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d23f0a5ff6d430282256e9bddc186b9 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:38.480852 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5053b5d32c554fc28514ad63bf81d9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.481488 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5053b5d32c554fc28514ad63bf81d9e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.483058 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 08dafb661b2443f5afadc6816868bac8 (DEFAULT_TABLE table=default.Table_72cbb579f8c5457ab8ef64876959a3a5 [id=8c16005dc18b44a4918959ca3868f9f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.486919 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.486850 12961 tablet_service.cc:1467] Processing CreateTablet for tablet a1160ab972124630a9be7abeefb4aa56 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.487473 12948 tablet_service.cc:1467] Processing CreateTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 (DEFAULT_TABLE table=default.Table_83626d3a5f3348679ec14bda1af29deb [id=034d601564f94db89efafdf7d3b35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.487879 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 08dafb661b2443f5afadc6816868bac8 (DEFAULT_TABLE table=default.Table_72cbb579f8c5457ab8ef64876959a3a5 [id=8c16005dc18b44a4918959ca3868f9f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.488094 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.488423 12964 tablet_service.cc:1467] Processing CreateTablet for tablet d7be9788df8e47d285bd932d078c6a8b (DEFAULT_TABLE table=default.Table_75c7fd329faa49bf99581532e6fd75d2 [id=8baacc24dda44dd5ad46aef596e96276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.488759 12959 tablet_service.cc:1467] Processing CreateTablet for tablet d9443cdbaa3c40299b66d216673c4b1a (DEFAULT_TABLE table=default.Table_1e8cbfa77d85462baa8b75f47a5c76bf [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.491910 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb (DEFAULT_TABLE table=default.Table_e75fdc7de2104d288d8d6bf0353108f3 [id=4884fa87524f4bdebb7c9e61f25a48a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.492108 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 (DEFAULT_TABLE table=default.Table_b666605912f04406bd8d879caa8c8b9b [id=85453d1f205a4fc5bcf2dd05063162f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.492722 12922 leader_election.cc:302] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.506389 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997ecba7fbae41178a25bf7af4d152e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.509084 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 163299us
I20240506 19:56:38.509457 12922 leader_election.cc:302] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.509759 13630 raft_consensus.cc:2781] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:38.509912 13630 raft_consensus.cc:482] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:38.510037 13630 raft_consensus.cc:3037] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:38.510480 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bf24d6ac33304466aa1b9cc642d29a57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.511248 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf24d6ac33304466aa1b9cc642d29a57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.516253 13433 raft_consensus.cc:482] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.516494 13433 raft_consensus.cc:504] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.517520 13433 leader_election.cc:288] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:38.518817 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:38.520092 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 116df9bd6ab04a30ba09861e765d7267 (DEFAULT_TABLE table=default.Table_9e78a3376c14485f9a70307f926edaa5 [id=db8bd7495bce4c518581981bca778450]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.528457 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9443cdbaa3c40299b66d216673c4b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.528630 13625 raft_consensus.cc:482] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.528846 13625 raft_consensus.cc:504] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.529318 13632 raft_consensus.cc:2726] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.529592 13625 leader_election.cc:288] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.532948 13625 raft_consensus.cc:482] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.533113 13625 raft_consensus.cc:504] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.533825 13625 leader_election.cc:288] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.534003 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dafb661b2443f5afadc6816868bac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.535993 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7be9788df8e47d285bd932d078c6a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.545341 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 116df9bd6ab04a30ba09861e765d7267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.545773 13546 tablet.cc:1722] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:38.551182 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1160ab972124630a9be7abeefb4aa56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.556711 13246 ts_tablet_manager.cc:1888] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:38.556886 13246 log.cc:1196] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1b7f56fea34946e590dae0621b28f9c6
I20240506 19:56:38.557309 13246 ts_tablet_manager.cc:1909] T 1b7f56fea34946e590dae0621b28f9c6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:38.561424 13246 tablet_replica.cc:330] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.561964 13246 raft_consensus.cc:2229] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:38.562366 13246 raft_consensus.cc:2258] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.563477 13246 ts_tablet_manager.cc:1875] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.566330 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1b7f56fea34946e590dae0621b28f9c6 (table default.Table_65407acf1df5470f8ec9e008fdbcbd93 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:38.571707 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06187c83b94d03ae4ebb67a2873cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.581473 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d1f01c0edb4f6b8a621c1643ca1afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.577651 13630 raft_consensus.cc:504] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.606922 13630 leader_election.cc:288] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:38.627604 13630 consensus_queue.cc:1035] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:38.628659 12922 leader_election.cc:341] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.632064 12781 leader_election.cc:302] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:38.634370 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 122727us
I20240506 19:56:38.637753 13517 raft_consensus.cc:2726] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.638444 12826 tablet_service.cc:1467] Processing CreateTablet for tablet a1160ab972124630a9be7abeefb4aa56 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.640146 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e3f798d6484d4fabd55f9b4c98a911" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:38.640273 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a080ccac6394210b0b24b65843f9d75" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:38.640961 12781 leader_election.cc:341] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d843fe82af1148f4a9045ce8b09ad11d
I20240506 19:56:38.641695 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:38.648424 13058 leader_election.cc:341] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.651793 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8a9ecf47d7474c82758417449d940c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:38.653613 13058 leader_election.cc:341] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.678227 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a080ccac6394210b0b24b65843f9d75" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:38.678586 13625 raft_consensus.cc:482] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.678778 13625 raft_consensus.cc:504] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.679507 13625 leader_election.cc:288] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.687153 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04738def56324a368f8182543107e32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.674453 12781 leader_election.cc:302] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:38.693542 13517 raft_consensus.cc:2726] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.697762 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ce7f4ceb9d40009e9c75543b1fc545. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.699378 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99f4861a01404663870ce91978c8af4f consensus state has no leader
I20240506 19:56:38.700791 12519 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]). Expected version 1 got 0
W20240506 19:56:38.705128 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 56308.9us
I20240506 19:56:38.713409 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997ecba7fbae41178a25bf7af4d152e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.721477 12781 leader_election.cc:341] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 30cb3cb7cbe14cc9bf1bfd07227f538e
I20240506 19:56:38.721659 12781 leader_election.cc:302] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:38.723438 13517 raft_consensus.cc:2726] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.723728 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d1f01c0edb4f6b8a621c1643ca1afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.726516 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17da01620b8b41008421b977455e0c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.728992 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeae70e0d1144ea9e3e0f8727c15e6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.729036 12921 leader_election.cc:341] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:38.729355 12921 leader_election.cc:341] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.730013 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6352d2c630204df3a24e299e167f92d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.730316 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dafb661b2443f5afadc6816868bac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.730682 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9443cdbaa3c40299b66d216673c4b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.731520 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1160ab972124630a9be7abeefb4aa56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.732321 12921 leader_election.cc:302] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.702136 13628 leader_election.cc:288] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:38.735903 13057 leader_election.cc:341] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.736102 13057 leader_election.cc:302] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.736577 13625 raft_consensus.cc:2726] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.739651 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8a9ecf47d7474c82758417449d940c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:38.739948 12988 raft_consensus.cc:2454] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:38.741493 13628 consensus_queue.cc:1035] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240506 19:56:38.742118 13630 raft_consensus.cc:2726] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.745597 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5516e8aad4e8474692b5b6c40bc483c1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:38.750595 12986 raft_consensus.cc:2454] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:38.752214 12922 leader_election.cc:341] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: c5e3f798d6484d4fabd55f9b4c98a911
I20240506 19:56:38.752480 12922 leader_election.cc:302] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.765630 13246 ts_tablet_manager.cc:1888] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:38.765808 13246 log.cc:1196] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/581aae0889654903933bef3bf1026954
I20240506 19:56:38.766180 13246 ts_tablet_manager.cc:1909] T 581aae0889654903933bef3bf1026954 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:38.770752 13246 tablet_replica.cc:330] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.771068 13246 raft_consensus.cc:2229] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:38.771307 13246 raft_consensus.cc:2258] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:38.771534 12922 consensus_peers.cc:485] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b7f56fea34946e590dae0621b28f9c6. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:38.772038 13057 leader_election.cc:302] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.772516 13035 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:38.772558 13057 leader_election.cc:302] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:38.772933 13625 raft_consensus.cc:2781] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:38.773058 13625 raft_consensus.cc:482] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:38.773135 13660 raft_consensus.cc:2781] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:38.773166 13625 raft_consensus.cc:3037] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:38.773238 13660 raft_consensus.cc:482] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:38.773376 13660 raft_consensus.cc:3037] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:38.776664 13628 raft_consensus.cc:2726] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:38.777621 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 581aae0889654903933bef3bf1026954 (table default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]) successfully deleted
I20240506 19:56:38.778223 13246 ts_tablet_manager.cc:1875] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.783216 13243 ts_tablet_manager.cc:1888] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:38.783526 13243 log.cc:1196] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/581aae0889654903933bef3bf1026954
I20240506 19:56:38.783970 13243 ts_tablet_manager.cc:1909] T 581aae0889654903933bef3bf1026954 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:38.788311 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 116df9bd6ab04a30ba09861e765d7267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.790616 12818 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 303) took 1039 ms. Trace:
I20240506 19:56:38.790802 12818 rpcz_store.cc:276] 0506 19:56:37.741123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.770700 (+ 29577us) service_pool.cc:225] Handling call
0506 19:56:37.807920 (+ 37220us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.807960 (+ 40us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:38.306238 (+498278us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.717954 (+411716us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:38.780456 (+ 62502us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 19:56:38.792207 13353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.020s sys 0.053s
W20240506 19:56:38.792357 13353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.854s user 0.020s sys 0.053s
I20240506 19:56:38.801936 12813 tablet_service.cc:1467] Processing CreateTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 (DEFAULT_TABLE table=default.Table_83626d3a5f3348679ec14bda1af29deb [id=034d601564f94db89efafdf7d3b35f05]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:38.804423 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c96f014293242be9640515697c0733b consensus state has no leader
I20240506 19:56:38.805132 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 17da01620b8b41008421b977455e0c9a
I20240506 19:56:38.806126 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:38.806120 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 17da01620b8b41008421b977455e0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.807062 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:38.807125 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.807838 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.813258 12818 tablet_service.cc:1467] Processing CreateTablet for tablet d7be9788df8e47d285bd932d078c6a8b (DEFAULT_TABLE table=default.Table_75c7fd329faa49bf99581532e6fd75d2 [id=8baacc24dda44dd5ad46aef596e96276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.813658 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fec5b763775747c9b4d63e7f0aed22ec" } new_table_name: "default.Table_23d6ef6eb15d489e813fdd5562e5be63" modify_external_catalogs: true
I20240506 19:56:38.795933 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5516e8aad4e8474692b5b6c40bc483c1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:38.814710 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 607b3fc2c17b4793af9b84c19066ec5e
I20240506 19:56:38.815467 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8b16088d777a4dcfb6b6d4be569babec" } new_table_name: "default.Table_77ba2f5ca8614d1ebb987a154041901c" modify_external_catalogs: true
I20240506 19:56:38.815691 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:38.815696 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b0f493ce68af4ecb9209de0432cb8184"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.816429 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0f493ce68af4ecb9209de0432cb8184 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.817386 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cae4c030228d49ff88f3c05320bd8924" } new_table_name: "default.Table_ea42f8ad6dc0468fba4b4876d9651595" modify_external_catalogs: true
I20240506 19:56:38.818717 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_feb73e349178483fb087f43a269605b0" } modify_external_catalogs: true
I20240506 19:56:38.818922 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_feb73e349178483fb087f43a269605b0" } modify_external_catalogs: true
W20240506 19:56:38.821489 13058 leader_election.cc:341] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.823664 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.825815 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.826238 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 17da01620b8b41008421b977455e0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.826562 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:38.827996 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_74372670a4e94691bb3c743dd7da48f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:38.828066 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fca8db6133de4be59cb24acf084124ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:38.828266 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ca95166c487c4067a47b74eef95deedc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.828606 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74372670a4e94691bb3c743dd7da48f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:38.828660 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca95166c487c4067a47b74eef95deedc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:38.828763 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fca8db6133de4be59cb24acf084124ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.832507 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_80e93d6f7329468c85d894226c0390c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.834796 13249 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.013s sys 0.014s
W20240506 19:56:38.833281 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80e93d6f7329468c85d894226c0390c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:38.842356 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2519c03738864f6ab5dcfab7bc3fc8aa consensus state has no leader
W20240506 19:56:38.841452 13249 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.529s user 0.013s sys 0.015s
I20240506 19:56:38.835433 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 393073bca2a04d15b3e583700262f3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b529bb9362924caca399491e7d9e03cd) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:38.837431 13629 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.017s sys 0.017s
W20240506 19:56:38.845919 13629 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.720s user 0.017s sys 0.017s
I20240506 19:56:38.850535 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff3435c11854649b7daf74f3d8c33d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.850719 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 17da01620b8b41008421b977455e0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:38.853132 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:38.854414 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:38.856057 13249 tablet_bootstrap.cc:492] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:38.856280 13249 ts_tablet_manager.cc:1367] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.574s user 0.016s sys 0.015s
I20240506 19:56:38.857264 13249 raft_consensus.cc:348] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.857573 13249 raft_consensus.cc:374] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:38.857659 13249 raft_consensus.cc:729] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:38.859915 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:38.860421 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7c61244620487bbbac26dcdbf8a38c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:38.867089 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6e54543d1544b1c8cff3e890e787337 consensus state has no leader
I20240506 19:56:38.867713 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_632a267fe67b4ac1af1a9882527ea33c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.868453 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_632a267fe67b4ac1af1a9882527ea33c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.872447 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4d65e20e3d924618a805ec2be5bb9b29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:38.873054 13246 ts_tablet_manager.cc:1888] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:38.873088 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d65e20e3d924618a805ec2be5bb9b29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:38.873204 13246 log.cc:1196] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2502b423bd1040539fb1a3ebbece3c1c
I20240506 19:56:38.873659 13246 ts_tablet_manager.cc:1909] T 2502b423bd1040539fb1a3ebbece3c1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:38.874382 13662 raft_consensus.cc:482] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.874337 13625 raft_consensus.cc:504] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.874554 13662 raft_consensus.cc:504] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.875094 13625 leader_election.cc:288] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.875662 13662 leader_election.cc:288] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.875797 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8a9ecf47d7474c82758417449d940c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:38.876025 12986 raft_consensus.cc:3037] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:38.878374 13242 ts_tablet_manager.cc:1888] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:38.878975 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd3fc62e9614cf3bde0202ee6832bb3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:38.879470 13057 leader_election.cc:341] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.879766 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 237) took 2701 ms. Trace:
I20240506 19:56:38.879920 13243 rpcz_store.cc:276] 0506 19:56:36.178236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.178293 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:37.633380 (+1455087us) ts_tablet_manager.cc:1120] Deleting tablet 581aae0889654903933bef3bf1026954
0506 19:56:37.633636 (+ 256us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:37.918621 (+284985us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.782959 (+864338us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.879704 (+ 96745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448396}
I20240506 19:56:38.880548 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 581aae0889654903933bef3bf1026954 (table default.Table_e3807d65b604472889ea0fa7e4b5e69b [id=493f88f528114491b898c4a1156712fc]) successfully deleted
I20240506 19:56:38.881078 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 247) took 2318 ms. Trace:
I20240506 19:56:38.881234 13243 rpcz_store.cc:276] 0506 19:56:36.562294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.566192 (+ 3898us) service_pool.cc:225] Handling call
0506 19:56:38.880894 (+2314702us) ts_tablet_manager.cc:1120] Deleting tablet 104337875517445f862d82ece3a84889
0506 19:56:38.880904 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.881027 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314471}
W20240506 19:56:38.881383 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4630abf54f024da5be5b45efccc0051d consensus state has no leader
W20240506 19:56:38.881839 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 104337875517445f862d82ece3a84889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104337875517445f862d82ece3a84889 already in progress: creating tablet
I20240506 19:56:38.882325 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 248) took 2310 ms. Trace:
I20240506 19:56:38.882522 13243 rpcz_store.cc:276] 0506 19:56:36.571936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.574489 (+ 2553us) service_pool.cc:225] Handling call
0506 19:56:38.882155 (+2307666us) ts_tablet_manager.cc:1120] Deleting tablet 4be929f0e72e4d22a9af2fe6b3072bbe
0506 19:56:38.882166 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.882267 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2307303}
I20240506 19:56:38.885370 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8a9ecf47d7474c82758417449d940c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:38.885308 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c9a95dafc84a43e5926a1f04aa925c19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:38.885964 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9a95dafc84a43e5926a1f04aa925c19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:38.886297 13058 leader_election.cc:341] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.889972 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 249) took 2292 ms. Trace:
I20240506 19:56:38.890218 13243 rpcz_store.cc:276] 0506 19:56:36.597022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.599286 (+ 2264us) service_pool.cc:225] Handling call
0506 19:56:38.889635 (+2290349us) ts_tablet_manager.cc:1120] Deleting tablet a36b4d51b100486c831a41a52d81fd66
0506 19:56:38.889650 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.889866 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2290060}
W20240506 19:56:38.890867 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a36b4d51b100486c831a41a52d81fd66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a36b4d51b100486c831a41a52d81fd66 already in progress: creating tablet
I20240506 19:56:38.891419 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 250) took 2234 ms. Trace:
I20240506 19:56:38.891580 13243 rpcz_store.cc:276] 0506 19:56:36.657005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.658151 (+ 1146us) service_pool.cc:225] Handling call
0506 19:56:38.891260 (+2233109us) ts_tablet_manager.cc:1120] Deleting tablet 17fb091762144f6591152680a6acd29e
0506 19:56:38.891269 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.891366 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232897}
W20240506 19:56:38.892108 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 17fb091762144f6591152680a6acd29e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17fb091762144f6591152680a6acd29e already in progress: creating tablet
I20240506 19:56:38.892478 13243 tablet_replica.cc:330] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:38.892789 13243 raft_consensus.cc:2229] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:38.893445 13243 raft_consensus.cc:2258] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.894241 13243 ts_tablet_manager.cc:1875] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:38.897032 13377 kernel_stack_watchdog.cc:198] Thread 13599 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 258ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f0f2b847980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f0f284b5d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f0f2f10d530 kudu::rpc::ReactorThread::Init() at ??:0
@ 0x7f0f2f118797 kudu::rpc::Reactor::Init() at ??:0
@ 0x7f0f2f0c15c5 kudu::rpc::Messenger::Init() at ??:0
@ 0x7f0f2f0c09d1 kudu::rpc::MessengerBuilder::Build() at ??:0
@ 0x7f0f3b7fbd8c kudu::master::AutoRebalancerTask::Init() at ??:0
@ 0x7f0f3b860dc2 kudu::master::CatalogManager::Init() at ??:0
@ 0x7f0f3ba0414f kudu::master::Master::InitCatalogManager() at ??:0
@ 0x7f0f3ba03ac1 kudu::master::Master::InitCatalogManagerTask() at ??:0
@ 0x7f0f3ba101d0 kudu::master::Master::StartAsync()::$_0::operator()() at ??:0
@ 0x7f0f3ba0fff9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240506 19:56:38.929950 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f0dfb5ea8de346149cc9b1f245fd94aa" } new_table_name: "default.Table_4adca98d503a4754b9f11888dc549378" modify_external_catalogs: true
I20240506 19:56:38.953814 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 452a83470585433e8def7174300d3a7f (DEFAULT_TABLE table=default.Table_2e36b2f9be7f4af9a58215847e671f2f [id=3a1dae7697ec4d529d8bb162d1170b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.954262 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452a83470585433e8def7174300d3a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.956951 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 40b9731a473449b19a9e95653a9d0830 (DEFAULT_TABLE table=default.Table_b40bb86f59744dc387a12a83d5dc8b9b [id=c4408f37ca81447585abb9f25a9f4a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.957610 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 673dc7c83f8748a09c41d51b42214aaf (DEFAULT_TABLE table=default.Table_b8e4c400be8f434295e3730c95d14641 [id=59da284ab7244c98b6627a0263be2c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.957696 13662 raft_consensus.cc:482] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:38.957847 13662 raft_consensus.cc:504] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.958060 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673dc7c83f8748a09c41d51b42214aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.958577 13662 leader_election.cc:288] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.961760 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 40b9731a473449b19a9e95653a9d0830 (DEFAULT_TABLE table=default.Table_b40bb86f59744dc387a12a83d5dc8b9b [id=c4408f37ca81447585abb9f25a9f4a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.962323 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b9731a473449b19a9e95653a9d0830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.962502 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2be78cbc8264f6e8d2e05137135161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.961539 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b24fa4663244a88dac9ea99fe38f3a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:38.964792 13057 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:38.965559 13242 log.cc:1196] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d0e3260f46024386baac64cb8677883c
I20240506 19:56:38.953785 13249 consensus_queue.cc:260] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.965766 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06187c83b94d03ae4ebb67a2873cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.965762 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (DEFAULT_TABLE table=default.Table_211f1d00bb6e407b90eac9c696f99e83 [id=a8b5f55150a54dcf8a19dc9e1153ea87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.966157 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4e97d10ae54e558920d116d8fa0ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.968786 13249 ts_tablet_manager.cc:1398] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.112s user 0.002s sys 0.002s
I20240506 19:56:38.969694 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 (DEFAULT_TABLE table=default.Table_c0d358df6dd84d60a343a335303508eb [id=b759e15bf3d341939e506cb620dd5448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.970220 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa8f9aaca444c839a88c07f58cb6ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.970274 12986 raft_consensus.cc:2454] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:38.971510 13249 tablet_bootstrap.cc:492] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:38.976562 13660 raft_consensus.cc:504] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.977092 13246 tablet_replica.cc:330] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:38.977346 13246 raft_consensus.cc:2229] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:38.977478 13660 leader_election.cc:288] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:38.977540 13246 raft_consensus.cc:2258] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:38.978199 13246 ts_tablet_manager.cc:1875] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:38.978622 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 452a83470585433e8def7174300d3a7f (DEFAULT_TABLE table=default.Table_2e36b2f9be7f4af9a58215847e671f2f [id=3a1dae7697ec4d529d8bb162d1170b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.979102 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452a83470585433e8def7174300d3a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.979615 13057 leader_election.cc:302] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:38.980104 13660 raft_consensus.cc:2781] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:38.980286 13660 raft_consensus.cc:686] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:38.980612 13660 consensus_queue.cc:237] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:38.981094 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2502b423bd1040539fb1a3ebbece3c1c (table default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]) successfully deleted
I20240506 19:56:38.981469 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8a9ecf47d7474c82758417449d940c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:38.981509 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f3e7329b22414da886dee4e1ec6bf3ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3e7329b22414da886dee4e1ec6bf3ad already in progress: creating tablet
I20240506 19:56:38.981716 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0d827d732e4261a7a8edb549c1c8e3 (DEFAULT_TABLE table=default.Table_27b96f0770c845b38361dfc9ac3f86f1 [id=25076b7bd89a4c3bbb205acaab09690a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.981914 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 (DEFAULT_TABLE table=default.Table_c0d358df6dd84d60a343a335303508eb [id=b759e15bf3d341939e506cb620dd5448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.982132 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0d827d732e4261a7a8edb549c1c8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.982331 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa8f9aaca444c839a88c07f58cb6ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.982911 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (DEFAULT_TABLE table=default.Table_211f1d00bb6e407b90eac9c696f99e83 [id=a8b5f55150a54dcf8a19dc9e1153ea87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.983275 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4e97d10ae54e558920d116d8fa0ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.985213 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 40b9731a473449b19a9e95653a9d0830 (DEFAULT_TABLE table=default.Table_b40bb86f59744dc387a12a83d5dc8b9b [id=c4408f37ca81447585abb9f25a9f4a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.985651 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 10ca9a1eaffc45f486d7a7e937a3e787 (DEFAULT_TABLE table=default.Table_e87268955e8247709b420f45ae2e4e5d [id=12a8e13309bc412ba8404129577b3444]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.985813 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0d827d732e4261a7a8edb549c1c8e3 (DEFAULT_TABLE table=default.Table_27b96f0770c845b38361dfc9ac3f86f1 [id=25076b7bd89a4c3bbb205acaab09690a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.986039 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ca9a1eaffc45f486d7a7e937a3e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.986295 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 673dc7c83f8748a09c41d51b42214aaf (DEFAULT_TABLE table=default.Table_b8e4c400be8f434295e3730c95d14641 [id=59da284ab7244c98b6627a0263be2c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.986353 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 10ca9a1eaffc45f486d7a7e937a3e787 (DEFAULT_TABLE table=default.Table_e87268955e8247709b420f45ae2e4e5d [id=12a8e13309bc412ba8404129577b3444]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:38.986778 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5516e8aad4e8474692b5b6c40bc483c1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:38.986984 12986 raft_consensus.cc:3037] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:38.988482 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b9731a473449b19a9e95653a9d0830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.994084 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ca9a1eaffc45f486d7a7e937a3e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.994462 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673dc7c83f8748a09c41d51b42214aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.994808 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0d827d732e4261a7a8edb549c1c8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:38.997486 13095 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 (DEFAULT_TABLE table=default.Table_e737c1fa78974f05a022649a900e2636 [id=e3446e2531334554961b65e81f9bb53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.001338 12949 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 (DEFAULT_TABLE table=default.Table_e737c1fa78974f05a022649a900e2636 [id=e3446e2531334554961b65e81f9bb53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.001958 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffb88e0a0c4125b04f4d8d328da3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.009460 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0e976f7b85cd40e49f346f31b8a5d1ee" } new_table_name: "default.Table_052f7d1eab54464d8743bd859c9f5bad" modify_external_catalogs: true
I20240506 19:56:39.010114 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5053b5d32c554fc28514ad63bf81d9e8" } modify_external_catalogs: true
I20240506 19:56:39.011525 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6974ae6398ae4559a50fb236133424b1" } modify_external_catalogs: true
I20240506 19:56:39.011593 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_75c7fd329faa49bf99581532e6fd75d2" } modify_external_catalogs: true
I20240506 19:56:39.016577 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6974ae6398ae4559a50fb236133424b1" } modify_external_catalogs: true
I20240506 19:56:39.016785 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_75c7fd329faa49bf99581532e6fd75d2" } modify_external_catalogs: true
I20240506 19:56:39.018458 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d909ad6f09e24952918c9471609c9d27" } modify_external_catalogs: true
I20240506 19:56:39.018676 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d909ad6f09e24952918c9471609c9d27" } modify_external_catalogs: true
I20240506 19:56:39.019946 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5053b5d32c554fc28514ad63bf81d9e8" } modify_external_catalogs: true
W20240506 19:56:39.016633 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f47449a5a504f939e131743d94433ee consensus state has no leader
W20240506 19:56:39.024925 13245 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.059s sys 0.159s
W20240506 19:56:39.025125 13245 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.549s user 0.059s sys 0.159s
I20240506 19:56:39.028465 13245 tablet_bootstrap.cc:492] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:39.028676 13245 ts_tablet_manager.cc:1367] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.561s user 0.062s sys 0.161s
I20240506 19:56:39.031316 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.031855 13353 raft_consensus.cc:482] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:39.031980 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.032073 13353 raft_consensus.cc:504] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.033430 13353 leader_election.cc:288] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:39.033656 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bf6a9df4bf204889a5a4737a2a61b964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.033880 13249 tablet_bootstrap.cc:654] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:39.034276 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf6a9df4bf204889a5a4737a2a61b964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:39.034780 13599 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.591s user 0.123s sys 0.200s
W20240506 19:56:39.034941 13599 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.591s user 0.123s sys 0.200s
I20240506 19:56:39.034404 13245 raft_consensus.cc:348] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.035527 13245 raft_consensus.cc:374] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:39.035666 13245 raft_consensus.cc:729] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:39.036600 13245 consensus_queue.cc:260] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.023958 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd3fc62e9614cf3bde0202ee6832bb3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:39.039088 12922 leader_election.cc:341] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2502b423bd1040539fb1a3ebbece3c1c
I20240506 19:56:39.043284 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 673dc7c83f8748a09c41d51b42214aaf (DEFAULT_TABLE table=default.Table_b8e4c400be8f434295e3730c95d14641 [id=59da284ab7244c98b6627a0263be2c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.045138 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bb95295253b14e0eaff2dceb4887634e" } new_table_name: "default.Table_e83de886956a4a2ba07ef464224044f6" modify_external_catalogs: true
I20240506 19:56:39.042994 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b24fa4663244a88dac9ea99fe38f3a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:39.047466 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b9731a473449b19a9e95653a9d0830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.048565 13249 tablet_bootstrap.cc:492] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:39.048734 13249 ts_tablet_manager.cc:1367] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20240506 19:56:39.049595 13249 raft_consensus.cc:348] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.049813 13249 raft_consensus.cc:374] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:39.049947 13249 raft_consensus.cc:729] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:39.050218 13249 consensus_queue.cc:260] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.051141 13245 ts_tablet_manager.cc:1398] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240506 19:56:39.051380 13249 ts_tablet_manager.cc:1398] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:39.051746 13249 tablet_bootstrap.cc:492] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:39.052035 13245 tablet_bootstrap.cc:492] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:39.053692 13249 tablet_bootstrap.cc:654] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:39.057461 13245 tablet_bootstrap.cc:654] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:39.051846 13058 leader_election.cc:341] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:39.061484 13058 leader_election.cc:302] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:39.062052 13662 raft_consensus.cc:2726] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:39.062314 13058 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:39.062475 13058 leader_election.cc:302] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:39.062743 13662 raft_consensus.cc:2726] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:39.070052 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_4adca98d503a4754b9f11888dc549378 [id=6fcb7b163e714ce7bf9e991838356b50], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f47449a5a504f939e131743d94433ee consensus state has no leader
I20240506 19:56:39.070590 12986 raft_consensus.cc:2454] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:39.076303 13057 leader_election.cc:302] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:39.076603 13057 raft_consensus.cc:2670] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:39.079655 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5516e8aad4e8474692b5b6c40bc483c1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:39.082304 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 452a83470585433e8def7174300d3a7f (DEFAULT_TABLE table=default.Table_2e36b2f9be7f4af9a58215847e671f2f [id=3a1dae7697ec4d529d8bb162d1170b3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.078883 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (DEFAULT_TABLE table=default.Table_211f1d00bb6e407b90eac9c696f99e83 [id=a8b5f55150a54dcf8a19dc9e1153ea87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.084182 13242 ts_tablet_manager.cc:1909] T d0e3260f46024386baac64cb8677883c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:39.085510 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffb88e0a0c4125b04f4d8d328da3f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:39.090085 13058 leader_election.cc:341] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:39.115514 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502b423bd1040539fb1a3ebbece3c1c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:39.123515 12921 leader_election.cc:341] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:39.123795 12921 leader_election.cc:302] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:39.124356 13538 raft_consensus.cc:2726] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:39.127780 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7be9788df8e47d285bd932d078c6a8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:39.155622 12846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.002s sys 0.011s
W20240506 19:56:39.155766 12846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.830s user 0.002s sys 0.011s
I20240506 19:56:39.175547 13246 ts_tablet_manager.cc:1888] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:39.175747 13246 log.cc:1196] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/5516e8aad4e8474692b5b6c40bc483c1
I20240506 19:56:39.176143 13246 ts_tablet_manager.cc:1909] T 5516e8aad4e8474692b5b6c40bc483c1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:39.184859 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452a83470585433e8def7174300d3a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.191217 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4e97d10ae54e558920d116d8fa0ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.206257 12519 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26] alter complete (version 1)
I20240506 19:56:39.208998 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1615f6167aae4dc096b9f563ab5e6215
I20240506 19:56:39.210179 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.212402 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1ba0602f38e64789a126e75fc115f5c2" } new_table_name: "default.Table_ac829ce2695e44849b1d44918904dc8a" modify_external_catalogs: true
I20240506 19:56:39.213050 13246 tablet_replica.cc:330] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:39.213248 13246 raft_consensus.cc:2229] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:39.213464 13246 raft_consensus.cc:2258] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:39.214006 13246 ts_tablet_manager.cc:1875] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:39.214368 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c with delete_type TABLET_DATA_DELETED (replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.216572 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d7be9788df8e47d285bd932d078c6a8b
I20240506 19:56:39.217433 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3ce0a12d16814dc9a767895fe39f1247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.218027 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ce0a12d16814dc9a767895fe39f1247 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.218096 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet d7be9788df8e47d285bd932d078c6a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.218477 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 5516e8aad4e8474692b5b6c40bc483c1 (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
W20240506 19:56:39.218762 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ceeae70e0d1144ea9e3e0f8727c15e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceeae70e0d1144ea9e3e0f8727c15e6b already in progress: creating tablet
I20240506 19:56:39.218894 13680 tablet.cc:1722] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:39.219535 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f8951bf4873e4382bcd42a52368fb376
I20240506 19:56:39.220379 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0f2e3be5e4274da5ac5270fced5e3d84" } new_table_name: "default.Table_bece2f866f8449ad8ef2f48007dc14ef" modify_external_catalogs: true
I20240506 19:56:39.220513 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet f8951bf4873e4382bcd42a52368fb376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.221174 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.221855 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 85bc7cfd006f4d4ca95541dab6873d32
I20240506 19:56:39.222647 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8fc0fb56e9d5488e95d9534f003454ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.223222 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fc0fb56e9d5488e95d9534f003454ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.223524 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.228787 12513 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6f06edf8576c455783057158a7e86f28
I20240506 19:56:39.231076 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:39.232650 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5ae6e15691764b5f979643fff861ff95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.233247 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ae6e15691764b5f979643fff861ff95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.236933 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 99dcc418075a4468a1d9a9887c007a20 (table default.Table_e83de886956a4a2ba07ef464224044f6 [id=0769cf069b8847faab1e5922412d544c]). Expected version 1 got 0
I20240506 19:56:39.238693 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.239527 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f2954b0b8209472b821fba70c582d658"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.240303 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2954b0b8209472b821fba70c582d658 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.238533 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet f8951bf4873e4382bcd42a52368fb376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.242780 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_347aa01b8f10416c820511fd1f4963e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.238416 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet d7be9788df8e47d285bd932d078c6a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:39.243317 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_347aa01b8f10416c820511fd1f4963e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:39.246630 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30ad527c3f2a4b53affa928e149db1df consensus state has no leader
I20240506 19:56:39.247354 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_521a41c69e284a2b95e880e463c496d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.248178 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_521a41c69e284a2b95e880e463c496d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.256870 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 99dcc418075a4468a1d9a9887c007a20 (table default.Table_e83de886956a4a2ba07ef464224044f6 [id=0769cf069b8847faab1e5922412d544c]). Expected version 1 got 0
I20240506 19:56:39.258515 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e9155e761b9240aaa949f69e037e6a2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.259316 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9155e761b9240aaa949f69e037e6a2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.262948 13683 tablet.cc:1722] T 99dcc418075a4468a1d9a9887c007a20 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:39.267208 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 393073bca2a04d15b3e583700262f3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b529bb9362924caca399491e7d9e03cd) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.268193 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_76271c0b874e41b29c550ad9d849d913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.269068 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76271c0b874e41b29c550ad9d849d913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.268965 13684 tablet.cc:1722] T 99dcc418075a4468a1d9a9887c007a20 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:39.280056 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673dc7c83f8748a09c41d51b42214aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.314184 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9ec35e469b0e4d8ca1557a818fd7bee7" } new_table_name: "default.Table_58c38385950d438c9f8fd0451fdc858d" modify_external_catalogs: true
W20240506 19:56:39.317205 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6352d2c630204df3a24e299e167f92d4 consensus state has no leader
W20240506 19:56:39.317883 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 consensus state has no leader
I20240506 19:56:39.318511 12814 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 323) took 1352 ms. Trace:
I20240506 19:56:39.318655 12814 rpcz_store.cc:276] 0506 19:56:37.965955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.966147 (+ 192us) service_pool.cc:225] Handling call
0506 19:56:37.970139 (+ 3992us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.970168 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:38.913801 (+943633us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.318240 (+404439us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.318419 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.320039 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 (DEFAULT_TABLE table=default.Table_c0d358df6dd84d60a343a335303508eb [id=b759e15bf3d341939e506cb620dd5448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.320596 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa8f9aaca444c839a88c07f58cb6ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.320642 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 99dcc418075a4468a1d9a9887c007a20 (table default.Table_e83de886956a4a2ba07ef464224044f6 [id=0769cf069b8847faab1e5922412d544c]). Expected version 1 got 0
I20240506 19:56:39.333706 13249 tablet_bootstrap.cc:492] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:39.334249 13249 ts_tablet_manager.cc:1367] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.283s user 0.026s sys 0.044s
I20240506 19:56:39.335286 13249 raft_consensus.cc:348] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.335594 13249 raft_consensus.cc:374] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:39.335672 13249 raft_consensus.cc:729] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:39.336007 13249 consensus_queue.cc:260] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.343485 13245 tablet_bootstrap.cc:492] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:39.343725 13245 ts_tablet_manager.cc:1367] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.292s user 0.028s sys 0.049s
I20240506 19:56:39.344537 13245 raft_consensus.cc:348] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.344785 13245 raft_consensus.cc:374] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:39.344875 13245 raft_consensus.cc:729] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:39.345152 13245 consensus_queue.cc:260] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.345785 13245 ts_tablet_manager.cc:1398] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:39.345965 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.346139 13249 ts_tablet_manager.cc:1398] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240506 19:56:39.346474 13249 tablet_bootstrap.cc:492] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:39.348258 13245 tablet_bootstrap.cc:492] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:39.349174 13662 raft_consensus.cc:482] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:39.349335 13662 raft_consensus.cc:504] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.350425 13662 leader_election.cc:288] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:39.350847 13171 heartbeater.cc:347] Connected to a master server at 127.12.48.253:36819
I20240506 19:56:39.351433 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:39.351683 12986 raft_consensus.cc:2454] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:39.352190 13057 leader_election.cc:302] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:39.354724 13662 raft_consensus.cc:2781] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:39.354849 13662 raft_consensus.cc:482] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:39.354956 13662 raft_consensus.cc:3037] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:39.357520 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:39.358258 13058 leader_election.cc:341] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:39.368381 13245 tablet_bootstrap.cc:654] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:39.378108 12810 consensus_meta.cc:287] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent flushing consensus metadata: real 0.652s user 0.003s sys 0.002s
I20240506 19:56:39.619721 12810 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 327) took 1535 ms. Trace:
I20240506 19:56:39.619875 12810 rpcz_store.cc:276] 0506 19:56:38.084667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.084728 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:38.099909 (+ 15181us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:38.099968 (+ 59us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:38.725336 (+625368us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.619487 (+894151us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.619673 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.620434 12810 tablet_service.cc:1467] Processing CreateTablet for tablet bff7050bbc2e47cf823d25ec10b89718 (DEFAULT_TABLE table=default.Table_c9a95dafc84a43e5926a1f04aa925c19 [id=5faa78277f6641439177ede005a94e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.620949 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff7050bbc2e47cf823d25ec10b89718. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:39.441175 13377 kernel_stack_watchdog.cc:198] Thread 13599 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 252ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f0f2b847980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f0f284b5d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f0f2f10d530 kudu::rpc::ReactorThread::Init() at ??:0
@ 0x7f0f2f118797 kudu::rpc::Reactor::Init() at ??:0
@ 0x7f0f2f0c15c5 kudu::rpc::Messenger::Init() at ??:0
@ 0x7f0f2f0c09d1 kudu::rpc::MessengerBuilder::Build() at ??:0
@ 0x7f0f3b83a15c kudu::master::AutoLeaderRebalancerTask::Init() at ??:0
@ 0x7f0f3b860f88 kudu::master::CatalogManager::Init() at ??:0
@ 0x7f0f3ba0414f kudu::master::Master::InitCatalogManager() at ??:0
@ 0x7f0f3ba03ac1 kudu::master::Master::InitCatalogManagerTask() at ??:0
@ 0x7f0f3ba101d0 kudu::master::Master::StartAsync()::$_0::operator()() at ??:0
@ 0x7f0f3ba0fff9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240506 19:56:39.376614 13245 tablet_bootstrap.cc:492] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:39.622284 13245 ts_tablet_manager.cc:1367] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.274s user 0.004s sys 0.000s
I20240506 19:56:39.623093 13245 raft_consensus.cc:348] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.623298 13245 raft_consensus.cc:374] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:39.623378 13245 raft_consensus.cc:729] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:39.623615 13245 consensus_queue.cc:260] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.378545 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 286) took 1720 ms. Trace:
I20240506 19:56:39.624176 13245 ts_tablet_manager.cc:1398] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:39.624042 13242 rpcz_store.cc:276] 0506 19:56:37.658297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.658354 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:37.950092 (+291738us) ts_tablet_manager.cc:1120] Deleting tablet d0e3260f46024386baac64cb8677883c
0506 19:56:37.950101 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:38.472168 (+522067us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:38.878232 (+406064us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.378481 (+500249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":291471}
I20240506 19:56:39.624450 13245 tablet_bootstrap.cc:492] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:39.624462 13242 tablet_replica.cc:330] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:39.624671 13242 raft_consensus.cc:2229] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:39.624850 13242 raft_consensus.cc:2258] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:39.372062 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 796628a0859c4511a8cb35e495dc8886 consensus state has no leader
I20240506 19:56:39.625403 13242 ts_tablet_manager.cc:1875] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:39.627478 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_347aa01b8f10416c820511fd1f4963e6" } new_table_name: "default.Table_a6c3a4c9339a480db1e49222fe7e0368" modify_external_catalogs: true
I20240506 19:56:39.370524 13249 tablet_bootstrap.cc:654] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:39.513845 13669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:39.514091 13505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36542 (request call id 0) took 2086 ms. Trace:
I20240506 19:56:39.384462 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0d827d732e4261a7a8edb549c1c8e3 (DEFAULT_TABLE table=default.Table_27b96f0770c845b38361dfc9ac3f86f1 [id=25076b7bd89a4c3bbb205acaab09690a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.629720 13669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:39.630262 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0d827d732e4261a7a8edb549c1c8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.387451 13246 ts_tablet_manager.cc:1888] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:39.633610 13246 log.cc:1196] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/4a080ccac6394210b0b24b65843f9d75
I20240506 19:56:39.633823 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d0e3260f46024386baac64cb8677883c (table default.Table_b4a1ca7c489b432f93a40d7be8d66723 [id=49a2f7ab61854519b4b0bb82fc7146f7]) successfully deleted
I20240506 19:56:39.634030 13246 ts_tablet_manager.cc:1909] T 4a080ccac6394210b0b24b65843f9d75 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:39.400803 12828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 322) took 1434 ms. Trace:
I20240506 19:56:39.634400 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 343) took 1161 ms. Trace:
I20240506 19:56:39.634469 12828 rpcz_store.cc:276] 0506 19:56:37.965852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.966119 (+ 267us) service_pool.cc:225] Handling call
0506 19:56:37.969134 (+ 3015us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.969156 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.234123 (+1264967us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.400555 (+166432us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.400739 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.634516 13246 rpcz_store.cc:276] 0506 19:56:38.472562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.472623 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:39.212971 (+740348us) ts_tablet_manager.cc:1120] Deleting tablet 4a080ccac6394210b0b24b65843f9d75
0506 19:56:39.212978 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:39.340924 (+127946us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.387269 (+ 46345us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.634362 (+247093us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":739390}
I20240506 19:56:39.634907 13246 tablet_replica.cc:330] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:39.635102 13246 raft_consensus.cc:2229] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:39.635172 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 46916e778bf34188959e1cfff084bffc (DEFAULT_TABLE table=default.Table_632a267fe67b4ac1af1a9882527ea33c [id=df889494c5f34752bf16bde1b5a80964]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.635357 13246 pending_rounds.cc:62] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:39.635430 13246 pending_rounds.cc:68] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744034227220480 op_type: NO_OP noop_request { }
I20240506 19:56:39.635553 13246 raft_consensus.cc:2866] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:39.635602 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46916e778bf34188959e1cfff084bffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.635668 13246 raft_consensus.cc:2258] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:39.384855 12515 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_e83de886956a4a2ba07ef464224044f6 [id=0769cf069b8847faab1e5922412d544c] alter complete (version 1)
I20240506 19:56:39.637003 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4af9a240535b494fbe65842e4ddc343b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.404742 12822 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 (DEFAULT_TABLE table=default.Table_e737c1fa78974f05a022649a900e2636 [id=e3446e2531334554961b65e81f9bb53c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:39.637614 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4af9a240535b494fbe65842e4ddc343b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.637717 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffb88e0a0c4125b04f4d8d328da3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.639153 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 4a080ccac6394210b0b24b65843f9d75 (table default.Table_bb529dd8d37f40449f27b0e07cbc93dc [id=4b9a2f4ea4b1460f89e8981e952e0d2d]) successfully deleted
I20240506 19:56:39.395465 12948 tablet_service.cc:1467] Processing CreateTablet for tablet c659d848fa34442fb605f1757571b5e7 (DEFAULT_TABLE table=default.Table_b0f493ce68af4ecb9209de0432cb8184 [id=ed9b9d65276145bfaa26ab5ccc785b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.639719 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c659d848fa34442fb605f1757571b5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.405556 12820 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 319) took 1440 ms. Trace:
I20240506 19:56:39.641697 12820 rpcz_store.cc:276] 0506 19:56:37.965537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.966226 (+ 689us) service_pool.cc:225] Handling call
0506 19:56:37.970831 (+ 4605us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.970862 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.023436 (+1052574us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.405322 (+381886us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.405494 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.642217 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 (DEFAULT_TABLE table=default.Table_ca95166c487c4067a47b74eef95deedc [id=4fa50790af954dc899b79de2f8539532]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.642625 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c440555cfe4195b3ae86e1f1eb6a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.643030 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a2eabb30d4154960a44eb50a8a87d. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:39.643299 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:39.387341 13662 raft_consensus.cc:504] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.394060 13095 tablet_service.cc:1467] Processing CreateTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 (DEFAULT_TABLE table=default.Table_4d65e20e3d924618a805ec2be5bb9b29 [id=d78c40589bf3432aa232674727a95dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.644131 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bece304e0ba048ecbb2d92aa71fdf707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.644384 13662 leader_election.cc:288] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:39.415767 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b666605912f04406bd8d879caa8c8b9b" } modify_external_catalogs: true
I20240506 19:56:39.644649 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b666605912f04406bd8d879caa8c8b9b" } modify_external_catalogs: true
I20240506 19:56:39.405845 12827 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 318) took 1440 ms. Trace:
I20240506 19:56:39.645407 12827 rpcz_store.cc:276] 0506 19:56:37.965347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.966249 (+ 902us) service_pool.cc:225] Handling call
0506 19:56:37.967764 (+ 1515us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.967797 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.006152 (+1038355us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.387045 (+380893us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.405788 (+ 18743us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.645920 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 4f382a89f94e4fdf9b8266d049635329 (DEFAULT_TABLE table=default.Table_80e93d6f7329468c85d894226c0390c4 [id=84394ad0c2d74138a9ed892d80d66574]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.396512 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 838d9e7986164ffd916d2d842dc61f6e (DEFAULT_TABLE table=default.Table_bf24d6ac33304466aa1b9cc642d29a57 [id=f424bbb43b2a47449f998c21dae05cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.646301 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f382a89f94e4fdf9b8266d049635329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.646396 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d9e7986164ffd916d2d842dc61f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.394327 13094 tablet_service.cc:1467] Processing CreateTablet for tablet c659d848fa34442fb605f1757571b5e7 (DEFAULT_TABLE table=default.Table_b0f493ce68af4ecb9209de0432cb8184 [id=ed9b9d65276145bfaa26ab5ccc785b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.406203 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:39.649708 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c659d848fa34442fb605f1757571b5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.650035 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 (DEFAULT_TABLE table=default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.650405 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d53f3daaa2949a2b6883435f031dc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.419492 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bece2f866f8449ad8ef2f48007dc14ef" } modify_external_catalogs: true
I20240506 19:56:39.652205 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bece2f866f8449ad8ef2f48007dc14ef" } modify_external_catalogs: true
I20240506 19:56:39.397249 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 10ca9a1eaffc45f486d7a7e937a3e787 (DEFAULT_TABLE table=default.Table_e87268955e8247709b420f45ae2e4e5d [id=12a8e13309bc412ba8404129577b3444]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.396824 12964 tablet_service.cc:1467] Processing CreateTablet for tablet c1ff7a3638004012b08c973192c055a5 (DEFAULT_TABLE table=default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc [id=0f37f31664bf47f1a74d2c82f8767eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.653645 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ca9a1eaffc45f486d7a7e937a3e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.422034 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_23d6ef6eb15d489e813fdd5562e5be63" } modify_external_catalogs: true
I20240506 19:56:39.653834 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ff7a3638004012b08c973192c055a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.653947 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_23d6ef6eb15d489e813fdd5562e5be63" } modify_external_catalogs: true
I20240506 19:56:39.425027 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8be586b55bf2436698bfeb2f94fb465d" } modify_external_catalogs: true
I20240506 19:56:39.656373 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8be586b55bf2436698bfeb2f94fb465d" } modify_external_catalogs: true
I20240506 19:56:39.424410 12809 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 328) took 1339 ms. Trace:
I20240506 19:56:39.395041 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 838d9e7986164ffd916d2d842dc61f6e (DEFAULT_TABLE table=default.Table_bf24d6ac33304466aa1b9cc642d29a57 [id=f424bbb43b2a47449f998c21dae05cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.430505 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a577b6ec9dc64f0f9be2a66998f740b5" } modify_external_catalogs: true
I20240506 19:56:39.657362 12809 rpcz_store.cc:276] 0506 19:56:38.085396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.085471 (+ 75us) service_pool.cc:225] Handling call
0506 19:56:38.314368 (+228897us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:38.342775 (+ 28407us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:38.929522 (+586747us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.409213 (+479691us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.424344 (+ 15131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.397109 12957 tablet_service.cc:1467] Processing CreateTablet for tablet f3a8531a1793404699532295164a3895 (DEFAULT_TABLE table=default.Table_8bb7368b19d441baa65305aa9e64514d [id=02d0d3a205854bb891a026ecd91c4be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.657613 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a577b6ec9dc64f0f9be2a66998f740b5" } modify_external_catalogs: true
I20240506 19:56:39.657886 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 45da091c44824bd6bb8d390a5fb01ef8 (DEFAULT_TABLE table=default.Table_e9155e761b9240aaa949f69e037e6a2b [id=22fff2da3af34564b794765f53773f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.657967 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a8531a1793404699532295164a3895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.658241 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da091c44824bd6bb8d390a5fb01ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.434970 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5344cd78a3dd4575a939a926d7e1eb16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.659107 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5344cd78a3dd4575a939a926d7e1eb16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.436430 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_521a41c69e284a2b95e880e463c496d8" } new_table_name: "default.Table_e9cb742b1de84e509d314bf1cc1444ed" modify_external_catalogs: true
I20240506 19:56:39.447978 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ce89088ef02b4c90bcfd607e38ba1608"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.395466 13090 tablet_service.cc:1467] Processing CreateTablet for tablet c1ff7a3638004012b08c973192c055a5 (DEFAULT_TABLE table=default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc [id=0f37f31664bf47f1a74d2c82f8767eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.426025 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet d7be9788df8e47d285bd932d078c6a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:39.661288 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce89088ef02b4c90bcfd607e38ba1608 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.397900 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 20aa06290b9a4814a006c423ccd8fc46 (DEFAULT_TABLE table=default.Table_fca8db6133de4be59cb24acf084124ef [id=75db5850c6074469bfd66e8d26699e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.661760 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20aa06290b9a4814a006c423ccd8fc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.661686 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 260d7ab458794df496012cb80466f59c (DEFAULT_TABLE table=default.Table_5ae6e15691764b5f979643fff861ff95 [id=be73b2a1315a45baa1db6feb44a3843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.662055 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260d7ab458794df496012cb80466f59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.663038 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:39.663275 12986 raft_consensus.cc:3037] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:39.396107 13097 tablet_service.cc:1467] Processing CreateTablet for tablet f3a8531a1793404699532295164a3895 (DEFAULT_TABLE table=default.Table_8bb7368b19d441baa65305aa9e64514d [id=02d0d3a205854bb891a026ecd91c4be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.396533 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 20aa06290b9a4814a006c423ccd8fc46 (DEFAULT_TABLE table=default.Table_fca8db6133de4be59cb24acf084124ef [id=75db5850c6074469bfd66e8d26699e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.397070 13104 tablet_service.cc:1467] Processing CreateTablet for tablet bff7050bbc2e47cf823d25ec10b89718 (DEFAULT_TABLE table=default.Table_c9a95dafc84a43e5926a1f04aa925c19 [id=5faa78277f6641439177ede005a94e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.397789 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 46916e778bf34188959e1cfff084bffc (DEFAULT_TABLE table=default.Table_632a267fe67b4ac1af1a9882527ea33c [id=df889494c5f34752bf16bde1b5a80964]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.398162 12949 tablet_service.cc:1467] Processing CreateTablet for tablet bff7050bbc2e47cf823d25ec10b89718 (DEFAULT_TABLE table=default.Table_c9a95dafc84a43e5926a1f04aa925c19 [id=5faa78277f6641439177ede005a94e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.400311 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 (DEFAULT_TABLE table=default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.426705 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet f8951bf4873e4382bcd42a52368fb376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:39.665583 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff7050bbc2e47cf823d25ec10b89718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.665714 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 458314e4653748e48cf646c72373fc3e (DEFAULT_TABLE table=default.Table_3ce0a12d16814dc9a767895fe39f1247 [id=55b6d1ff222b47a7831fce79b949314c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.666079 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458314e4653748e48cf646c72373fc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.398762 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 46916e778bf34188959e1cfff084bffc (DEFAULT_TABLE table=default.Table_632a267fe67b4ac1af1a9882527ea33c [id=df889494c5f34752bf16bde1b5a80964]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.669483 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:39.669677 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46916e778bf34188959e1cfff084bffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.401116 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 (DEFAULT_TABLE table=default.Table_ca95166c487c4067a47b74eef95deedc [id=4fa50790af954dc899b79de2f8539532]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.434517 12816 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 321) took 1468 ms. Trace:
I20240506 19:56:39.670017 12816 rpcz_store.cc:276] 0506 19:56:37.965757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.966096 (+ 339us) service_pool.cc:225] Handling call
0506 19:56:37.968262 (+ 2166us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.968285 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.126188 (+1157903us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.418618 (+292430us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.434459 (+ 15841us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:39.670511 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 1750c4dd8cdc40509c553b63d166c466 (DEFAULT_TABLE table=default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410 [id=1f5d46e4e6cb45e0ad944590ca94dd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.670929 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1750c4dd8cdc40509c553b63d166c466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.399325 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 (DEFAULT_TABLE table=default.Table_ca95166c487c4067a47b74eef95deedc [id=4fa50790af954dc899b79de2f8539532]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.673350 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c440555cfe4195b3ae86e1f1eb6a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.401454 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 4f382a89f94e4fdf9b8266d049635329 (DEFAULT_TABLE table=default.Table_80e93d6f7329468c85d894226c0390c4 [id=84394ad0c2d74138a9ed892d80d66574]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.438836 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:39.673945 12813 tablet_service.cc:1467] Processing CreateTablet for tablet f759b560e2ab41a6a076976ccca811d1 (DEFAULT_TABLE table=default.Table_bf6a9df4bf204889a5a4737a2a61b964 [id=0effc08192b244ce91035097b30895a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.674340 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f759b560e2ab41a6a076976ccca811d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.439692 12826 tablet_service.cc:1467] Processing CreateTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 (DEFAULT_TABLE table=default.Table_4d65e20e3d924618a805ec2be5bb9b29 [id=d78c40589bf3432aa232674727a95dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.675376 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bece304e0ba048ecbb2d92aa71fdf707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.400113 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 4f382a89f94e4fdf9b8266d049635329 (DEFAULT_TABLE table=default.Table_80e93d6f7329468c85d894226c0390c4 [id=84394ad0c2d74138a9ed892d80d66574]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.677230 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f382a89f94e4fdf9b8266d049635329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.432523 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 45da091c44824bd6bb8d390a5fb01ef8 (DEFAULT_TABLE table=default.Table_e9155e761b9240aaa949f69e037e6a2b [id=22fff2da3af34564b794765f53773f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.400978 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 (DEFAULT_TABLE table=default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.401271 12955 tablet_service.cc:1467] Processing CreateTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 (DEFAULT_TABLE table=default.Table_4d65e20e3d924618a805ec2be5bb9b29 [id=d78c40589bf3432aa232674727a95dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.435405 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 260d7ab458794df496012cb80466f59c (DEFAULT_TABLE table=default.Table_5ae6e15691764b5f979643fff861ff95 [id=be73b2a1315a45baa1db6feb44a3843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.434340 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 45da091c44824bd6bb8d390a5fb01ef8 (DEFAULT_TABLE table=default.Table_e9155e761b9240aaa949f69e037e6a2b [id=22fff2da3af34564b794765f53773f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.437475 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 458314e4653748e48cf646c72373fc3e (DEFAULT_TABLE table=default.Table_3ce0a12d16814dc9a767895fe39f1247 [id=55b6d1ff222b47a7831fce79b949314c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.436370 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 260d7ab458794df496012cb80466f59c (DEFAULT_TABLE table=default.Table_5ae6e15691764b5f979643fff861ff95 [id=be73b2a1315a45baa1db6feb44a3843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.440517 12821 tablet_service.cc:1467] Processing CreateTablet for tablet c659d848fa34442fb605f1757571b5e7 (DEFAULT_TABLE table=default.Table_b0f493ce68af4ecb9209de0432cb8184 [id=ed9b9d65276145bfaa26ab5ccc785b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.440125 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 1750c4dd8cdc40509c553b63d166c466 (DEFAULT_TABLE table=default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410 [id=1f5d46e4e6cb45e0ad944590ca94dd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.439102 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 458314e4653748e48cf646c72373fc3e (DEFAULT_TABLE table=default.Table_3ce0a12d16814dc9a767895fe39f1247 [id=55b6d1ff222b47a7831fce79b949314c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.678339 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c659d848fa34442fb605f1757571b5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.442835 13102 tablet_service.cc:1467] Processing CreateTablet for tablet f759b560e2ab41a6a076976ccca811d1 (DEFAULT_TABLE table=default.Table_bf6a9df4bf204889a5a4737a2a61b964 [id=0effc08192b244ce91035097b30895a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.441649 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 1750c4dd8cdc40509c553b63d166c466 (DEFAULT_TABLE table=default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410 [id=1f5d46e4e6cb45e0ad944590ca94dd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.678488 13669 leader_election.cc:288] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:39.446342 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 6ace0ec43ba44060bcec2a50e1f48098 (DEFAULT_TABLE table=default.Table_521a41c69e284a2b95e880e463c496d8 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.445152 12956 tablet_service.cc:1467] Processing CreateTablet for tablet f759b560e2ab41a6a076976ccca811d1 (DEFAULT_TABLE table=default.Table_bf6a9df4bf204889a5a4737a2a61b964 [id=0effc08192b244ce91035097b30895a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.448668 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 880ecc75bab446e39c687f3c8b740361 (DEFAULT_TABLE table=default.Table_347aa01b8f10416c820511fd1f4963e6 [id=0b3a486ff6d24c8795de3e53e83021a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.446939 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 6ace0ec43ba44060bcec2a50e1f48098 (DEFAULT_TABLE table=default.Table_521a41c69e284a2b95e880e463c496d8 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.451565 13086 tablet_service.cc:1467] Processing CreateTablet for tablet d43da1d12ce647dcbcd31263932b2f54 (DEFAULT_TABLE table=default.Table_8fc0fb56e9d5488e95d9534f003454ec [id=0b3eb5aa7ebf44948d41cb63f9ea6a9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.450177 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 880ecc75bab446e39c687f3c8b740361 (DEFAULT_TABLE table=default.Table_347aa01b8f10416c820511fd1f4963e6 [id=0b3a486ff6d24c8795de3e53e83021a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.453505 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 7b803ae289344f88b604a2053881d59b (DEFAULT_TABLE table=default.Table_f2954b0b8209472b821fba70c582d658 [id=d2ffd5634f2042b7a85cbe74d6ca1df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.452152 12950 tablet_service.cc:1467] Processing CreateTablet for tablet d43da1d12ce647dcbcd31263932b2f54 (DEFAULT_TABLE table=default.Table_8fc0fb56e9d5488e95d9534f003454ec [id=0b3eb5aa7ebf44948d41cb63f9ea6a9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.455152 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 7b803ae289344f88b604a2053881d59b (DEFAULT_TABLE table=default.Table_f2954b0b8209472b821fba70c582d658 [id=d2ffd5634f2042b7a85cbe74d6ca1df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.489158 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41490 (request call id 301) took 1779 ms. Trace:
I20240506 19:56:39.495640 13243 ts_tablet_manager.cc:1888] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:39.680061 13243 log.cc:1196] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/81ea09efd4aa4019911dd23774f464a2
I20240506 19:56:39.680598 13243 ts_tablet_manager.cc:1909] T 81ea09efd4aa4019911dd23774f464a2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:39.680037 13680 rpcz_store.cc:276] 0506 19:56:37.709611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.710193 (+ 582us) service_pool.cc:225] Handling call
0506 19:56:39.488713 (+1778520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:37.710651 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0506 19:56:37.710744 (+ 93us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0506 19:56:37.710753 (+ 9us) alter_schema_op.cc:59] Acquired schema lock
0506 19:56:37.710760 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0506 19:56:37.710798 (+ 38us) alter_schema_op.cc:119] START. Timestamp: 1715025397710678
0506 19:56:37.710801 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:37.710975 (+ 174us) log.cc:842] Serialized 173 byte log entry
0506 19:56:38.849714 (+1138739us) op_driver.cc:444] REPLICATION: finished
0506 19:56:39.218821 (+369107us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0506 19:56:39.219368 (+ 547us) log.cc:842] Serialized 5 byte log entry
0506 19:56:39.470701 (+251333us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.488287 (+ 17586us) log.cc:842] Serialized 14 byte log entry
0506 19:56:39.488684 (+ 397us) alter_schema_op.cc:65] Released schema lock
0506 19:56:39.488688 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":369003,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":426,"prepare.run_wall_time_us":427,"raft.queue_time_us":377,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":346,"replication_time_us":1138856,"thread_start_us":369941,"threads_started":1}]]}
I20240506 19:56:39.442384 12815 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 320) took 1476 ms. Trace:
I20240506 19:56:39.682533 12815 rpcz_store.cc:276] 0506 19:56:37.965655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.965999 (+ 344us) service_pool.cc:225] Handling call
0506 19:56:37.971763 (+ 5764us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:37.971783 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.187660 (+1215877us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.442169 (+254509us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:39.442327 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7790}
I20240506 19:56:39.683208 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 6ace0ec43ba44060bcec2a50e1f48098 (DEFAULT_TABLE table=default.Table_521a41c69e284a2b95e880e463c496d8 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.683676 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ace0ec43ba44060bcec2a50e1f48098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.685460 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880ecc75bab446e39c687f3c8b740361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.563922 13353 raft_consensus.cc:482] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:39.686446 13353 raft_consensus.cc:504] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.687405 13353 leader_election.cc:288] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:39.447116 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 838d9e7986164ffd916d2d842dc61f6e (DEFAULT_TABLE table=default.Table_bf24d6ac33304466aa1b9cc642d29a57 [id=f424bbb43b2a47449f998c21dae05cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.465389 12811 tablet_service.cc:1467] Processing CreateTablet for tablet c1ff7a3638004012b08c973192c055a5 (DEFAULT_TABLE table=default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc [id=0f37f31664bf47f1a74d2c82f8767eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.466298 13244 tablet_bootstrap.cc:654] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:39.467612 13689 tablet.cc:1722] T 99dcc418075a4468a1d9a9887c007a20 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:39.474866 12806 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 88) took 1563 ms. Trace:
I20240506 19:56:39.504307 12804 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 89) took 1543 ms. Trace:
I20240506 19:56:39.505802 12819 tablet_service.cc:1467] Processing CreateTablet for tablet f3a8531a1793404699532295164a3895 (DEFAULT_TABLE table=default.Table_8bb7368b19d441baa65305aa9e64514d [id=02d0d3a205854bb891a026ecd91c4be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.506843 12808 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 86) took 1804 ms. Trace:
I20240506 19:56:39.517436 12805 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 90) took 1391 ms. Trace:
I20240506 19:56:39.517688 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 20aa06290b9a4814a006c423ccd8fc46 (DEFAULT_TABLE table=default.Table_fca8db6133de4be59cb24acf084124ef [id=75db5850c6074469bfd66e8d26699e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.689478 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bece304e0ba048ecbb2d92aa71fdf707. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.693548 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da091c44824bd6bb8d390a5fb01ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.697522 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260d7ab458794df496012cb80466f59c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:39.698541 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 880ecc75bab446e39c687f3c8b740361 consensus state has no leader
I20240506 19:56:39.699091 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d9e7986164ffd916d2d842dc61f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.701452 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f759b560e2ab41a6a076976ccca811d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.705056 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d53f3daaa2949a2b6883435f031dc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.708935 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43da1d12ce647dcbcd31263932b2f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.712929 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1750c4dd8cdc40509c553b63d166c466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.715534 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:39.716703 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458314e4653748e48cf646c72373fc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.718230 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 320e714ed6b64f2d84770e7182e4103a
I20240506 19:56:39.719053 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d95065af2929456c8ae9740f3fcffcdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:39.719676 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d95065af2929456c8ae9740f3fcffcdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.719972 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b803ae289344f88b604a2053881d59b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:39.721165 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ace0ec43ba44060bcec2a50e1f48098 consensus state has no leader
I20240506 19:56:39.721694 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_604f6c5cb5214c97b981184d3202d345"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.722167 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4095f9fc28ff48f68064ec8318ff7b51
I20240506 19:56:39.722630 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f29103e88f944dbf883f0de41a88c053" } new_table_name: "default.Table_715977f202b24ca7a07c82ef612e23bb" modify_external_catalogs: true
I20240506 19:56:39.722908 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2f78de17b14b4a3c85876eb23bee958a" } new_table_name: "default.Table_35348a0401e440239d4fa0424c80bd0f" modify_external_catalogs: true
I20240506 19:56:39.723275 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ace0ec43ba44060bcec2a50e1f48098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.723984 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 997ecba7fbae41178a25bf7af4d152e1
I20240506 19:56:39.737571 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d9e7986164ffd916d2d842dc61f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.741777 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a8531a1793404699532295164a3895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.980345 13711 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240506 19:56:39.990073 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 328 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:39.745605 13249 tablet_bootstrap.cc:492] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.261807 13249 ts_tablet_manager.cc:1367] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.915s user 0.006s sys 0.010s
I20240506 19:56:40.262821 13249 raft_consensus.cc:348] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.263082 13249 raft_consensus.cc:374] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.263177 13249 raft_consensus.cc:729] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.263438 13249 consensus_queue.cc:260] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.264066 13249 ts_tablet_manager.cc:1398] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:40.264416 13249 tablet_bootstrap.cc:492] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:39.742372 13245 tablet_bootstrap.cc:654] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:39.743391 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ff7a3638004012b08c973192c055a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.724653 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c0ebfdd5c1cb4085babb98af4665cf5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.749387 13246 ts_tablet_manager.cc:1875] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:40.266227 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0ebfdd5c1cb4085babb98af4665cf5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.738323 12806 rpcz_store.cc:276] 0506 19:56:37.910622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.910688 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:38.190962 (+280274us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x616000111530. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4e891628 00007f7b5128f768 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:39.449951 (+1258989us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x616000141454. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4d57db00 00007f7b5128f95f 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:39.473553 (+ 23602us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57651456}
I20240506 19:56:39.740450 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20aa06290b9a4814a006c423ccd8fc46. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:40.136759 13442 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.48.254:33495 timed out after 0.502s (SENT)
I20240506 19:56:39.738111 12805 rpcz_store.cc:276] 0506 19:56:38.118452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.118524 (+ 72us) service_pool.cc:225] Handling call
0506 19:56:39.491934 (+1373410us) spinlock_profiling.cc:243] Waited 5.31 ms on lock 0x616000141454. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4d57db00 00007f7b5128f95f 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:39.509442 (+ 17508us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12205824}
I20240506 19:56:39.749147 12808 rpcz_store.cc:276] 0506 19:56:37.702602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.754089 (+ 51487us) service_pool.cc:225] Handling call
0506 19:56:38.119202 (+365113us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x6160001b0a54. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4d57db00 00007f7b5128f95f 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:39.506792 (+1387590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":519009536}
W20240506 19:56:40.268029 13442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.254:33495, user_credentials={real_user=slave}} blocked reactor thread for 131305us
I20240506 19:56:40.209380 12672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 328 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:39.740514 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd3fc62e9614cf3bde0202ee6832bb3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:39.832886 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 251) took 3153 ms. Trace:
I20240506 19:56:39.725364 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2486dda60f0b4f2b9a63dbf17885bc5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.889729 12986 raft_consensus.cc:2454] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:39.700603 12804 rpcz_store.cc:276] 0506 19:56:37.960309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.960372 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:39.350784 (+1390412us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x616000153154. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4d57db00 00007f7b5128f95f 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:39.503449 (+152665us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":129050496}
W20240506 19:56:40.269235 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2486dda60f0b4f2b9a63dbf17885bc5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:39.893982 13353 raft_consensus.cc:482] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:39.798457 13662 raft_consensus.cc:482] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:39.852267 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 50ef5a2799b447f992207452dbe39645 (DEFAULT_TABLE table=default.Table_76271c0b874e41b29c550ad9d849d913 [id=9d0bea602ade4b3cb1ae8d46915d4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.783344 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 880ecc75bab446e39c687f3c8b740361 (DEFAULT_TABLE table=default.Table_347aa01b8f10416c820511fd1f4963e6 [id=0b3a486ff6d24c8795de3e53e83021a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.896684 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 50ef5a2799b447f992207452dbe39645 (DEFAULT_TABLE table=default.Table_76271c0b874e41b29c550ad9d849d913 [id=9d0bea602ade4b3cb1ae8d46915d4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:39.748638 13058 leader_election.cc:341] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:40.270892 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 600595us
I20240506 19:56:40.271198 13057 leader_election.cc:302] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:39.787376 12820 tablet_service.cc:1467] Processing CreateTablet for tablet d43da1d12ce647dcbcd31263932b2f54 (DEFAULT_TABLE table=default.Table_8fc0fb56e9d5488e95d9534f003454ec [id=0b3eb5aa7ebf44948d41cb63f9ea6a9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.897014 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.864298 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2365d00044c32a06f9017a207adf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.834439 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 50ef5a2799b447f992207452dbe39645 (DEFAULT_TABLE table=default.Table_76271c0b874e41b29c550ad9d849d913 [id=9d0bea602ade4b3cb1ae8d46915d4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.272143 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.272362 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:39.897590 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 320e714ed6b64f2d84770e7182e4103a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.858530 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.272784 13093 tablet_service.cc:1467] Processing CreateTablet for tablet ddae4b0d8d0943d99f54904266f293d4 (DEFAULT_TABLE table=default.Table_ce89088ef02b4c90bcfd607e38ba1608 [id=5f422f6d344c45028cec3ca6f401bf2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.272886 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 320e714ed6b64f2d84770e7182e4103a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.273118 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:39.862638 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 7b803ae289344f88b604a2053881d59b (DEFAULT_TABLE table=default.Table_f2954b0b8209472b821fba70c582d658 [id=d2ffd5634f2042b7a85cbe74d6ca1df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.273236 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd3fc62e9614cf3bde0202ee6832bb3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:39.879551 13244 tablet_bootstrap.cc:492] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:40.273474 13124 raft_consensus.cc:2454] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:40.273531 13244 ts_tablet_manager.cc:1367] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 2.351s user 0.003s sys 0.015s
I20240506 19:56:39.725920 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_86f15af99695455a9f33070a0bc8563f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:39.915726 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:40.274503 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86f15af99695455a9f33070a0bc8563f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.274577 13094 tablet_service.cc:1467] Processing CreateTablet for tablet e1b011e94e2b47d7a51d6816576ea124 (DEFAULT_TABLE table=default.Table_4af9a240535b494fbe65842e4ddc343b [id=e8385d8ad6714acf91d8cb6b0de6469c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.274523 13244 raft_consensus.cc:348] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.274744 13244 raft_consensus.cc:374] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:39.950348 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 320e714ed6b64f2d84770e7182e4103a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.275082 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.235960 13722 raft_consensus.cc:482] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.275406 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 373e04e93a5543f9934077006eaf2ba9 (DEFAULT_TABLE table=default.Table_5344cd78a3dd4575a939a926d7e1eb16 [id=d6d66e91fd3540d292e12bfeac68617a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.275490 13244 raft_consensus.cc:729] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.275568 13722 raft_consensus.cc:504] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.275769 13244 consensus_queue.cc:260] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.276355 13722 leader_election.cc:288] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.276464 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:39.900533 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.276937 12812 tablet_service.cc:1467] Processing CreateTablet for tablet ddae4b0d8d0943d99f54904266f293d4 (DEFAULT_TABLE table=default.Table_ce89088ef02b4c90bcfd607e38ba1608 [id=5f422f6d344c45028cec3ca6f401bf2b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:39.722188 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_604f6c5cb5214c97b981184d3202d345 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.277130 13722 raft_consensus.cc:2781] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:39.725221 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8bea4c0b88b344d6ba5cf80917534deb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:40.277323 13722 raft_consensus.cc:686] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
W20240506 19:56:40.277629 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bea4c0b88b344d6ba5cf80917534deb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.277601 13722 consensus_queue.cc:237] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:39.901576 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:39.903649 12961 tablet_service.cc:1467] Processing CreateTablet for tablet ddae4b0d8d0943d99f54904266f293d4 (DEFAULT_TABLE table=default.Table_ce89088ef02b4c90bcfd607e38ba1608 [id=5f422f6d344c45028cec3ca6f401bf2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.904639 12952 tablet_service.cc:1467] Processing CreateTablet for tablet e1b011e94e2b47d7a51d6816576ea124 (DEFAULT_TABLE table=default.Table_4af9a240535b494fbe65842e4ddc343b [id=e8385d8ad6714acf91d8cb6b0de6469c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.905827 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 373e04e93a5543f9934077006eaf2ba9 (DEFAULT_TABLE table=default.Table_5344cd78a3dd4575a939a926d7e1eb16 [id=d6d66e91fd3540d292e12bfeac68617a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.726922 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2519c03738864f6ab5dcfab7bc3fc8aa
I20240506 19:56:39.990073 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 59b022c086d141a19a9b35c4a8c8bef9 (DEFAULT_TABLE table=default.Table_d95065af2929456c8ae9740f3fcffcdc [id=855c174fb8224b5a8acd4c0dd4b4f8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:39.765760 12532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 328 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
W20240506 19:56:40.279553 12532 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47760 (request call id 0) took 513 ms (client timeout 502 ms). Trace:
I20240506 19:56:39.803072 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d23f0a5ff6d430282256e9bddc186b9 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:40.281944 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b53ae7ee595f4d70813b67094f238eca" } new_table_name: "default.Table_ff179844ac74496183bb76ddcabbcc75" modify_external_catalogs: true
W20240506 19:56:39.803524 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b3268094cbf481698cb373c1bf6dfba consensus state has no leader
I20240506 19:56:40.283887 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_44e570af6caa4a749207a8c828639acf" } modify_external_catalogs: true
I20240506 19:56:40.284096 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_44e570af6caa4a749207a8c828639acf" } modify_external_catalogs: true
W20240506 19:56:39.875790 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2a5a5e193f245de96cc82986e292f05 consensus state has no leader
I20240506 19:56:40.286244 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880ecc75bab446e39c687f3c8b740361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:39.887287 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_25774424d24c4cd68a59fd89cae9b492"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:40.294703 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25774424d24c4cd68a59fd89cae9b492 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:40.070174 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet b2a5a5e193f245de96cc82986e292f05 (table default.Table_35348a0401e440239d4fa0424c80bd0f [id=f83571d593d64e58876888bfb31ccb28], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b2a5a5e193f245de96cc82986e292f05 consensus state has no leader
I20240506 19:56:40.300271 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fa840bd001a04582ac72848588e27126"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:40.300889 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa840bd001a04582ac72848588e27126 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.305485 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a62a1c7b556341a88bc49f8caa3c8238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:40.306146 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a62a1c7b556341a88bc49f8caa3c8238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.293983 12531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 328 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:40.309830 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a8531a1793404699532295164a3895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.310142 13442 leader_election.cc:302] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27d4482cd59a4e5687992b1f8a177373; no voters: 190a2eabb30d4154960a44eb50a8a87d, 97f390a786034cddbff757779037f861
W20240506 19:56:40.279697 12532 rpcz_store.cc:269] 0506 19:56:39.765584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.765643 (+ 59us) service_pool.cc:225] Handling call
0506 19:56:40.279486 (+513843us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.314718 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bf24d6ac33304466aa1b9cc642d29a57" } new_table_name: "default.Table_b41d42bb850b4a5b9b99fcacd93be3fb" modify_external_catalogs: true
I20240506 19:56:40.316244 13725 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:40.319888 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.320278 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ff7a3638004012b08c973192c055a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.320430 13244 ts_tablet_manager.cc:1398] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20240506 19:56:40.325395 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ef5a2799b447f992207452dbe39645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.311410 13243 rpcz_store.cc:276] 0506 19:56:36.679039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.679110 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:38.892390 (+2213280us) ts_tablet_manager.cc:1120] Deleting tablet 81ea09efd4aa4019911dd23774f464a2
0506 19:56:38.892400 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:39.339975 (+447575us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.489417 (+149442us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.832818 (+343401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213036}
I20240506 19:56:40.327613 13242 ts_tablet_manager.cc:1888] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.327754 13242 log.cc:1196] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d843fe82af1148f4a9045ce8b09ad11d
I20240506 19:56:40.327821 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 81ea09efd4aa4019911dd23774f464a2 (table default.Table_7af138f7bd89452c9f067f9c2fc6f955 [id=534590fb66294863bce3ed42ea46eb9a]) successfully deleted
I20240506 19:56:40.328148 13242 ts_tablet_manager.cc:1909] T d843fe82af1148f4a9045ce8b09ad11d P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:40.338478 12922 leader_election.cc:302] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:40.338752 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 63367.5us
I20240506 19:56:40.339443 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46916e778bf34188959e1cfff084bffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.340807 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c440555cfe4195b3ae86e1f1eb6a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.341517 13629 raft_consensus.cc:2781] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:40.341662 13629 raft_consensus.cc:482] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:40.340466 13662 raft_consensus.cc:504] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.347221 13629 raft_consensus.cc:3037] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.349792 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 252) took 3649 ms. Trace:
I20240506 19:56:40.350142 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b022c086d141a19a9b35c4a8c8bef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.351100 13243 rpcz_store.cc:276] 0506 19:56:36.687239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.687319 (+ 80us) service_pool.cc:225] Handling call
0506 19:56:40.327576 (+3640257us) ts_tablet_manager.cc:1120] Deleting tablet 618c56ef126a4ca48b6a379a6771bc1c
0506 19:56:40.327588 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.336557 (+ 8969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3639888}
I20240506 19:56:40.351764 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 253) took 3635 ms. Trace:
W20240506 19:56:40.350083 12921 leader_election.cc:341] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:40.353129 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 618c56ef126a4ca48b6a379a6771bc1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 618c56ef126a4ca48b6a379a6771bc1c already in progress: creating tablet
I20240506 19:56:40.355046 13662 leader_election.cc:288] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.356340 13243 rpcz_store.cc:276] 0506 19:56:36.716487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.716813 (+ 326us) service_pool.cc:225] Handling call
0506 19:56:40.351517 (+3634704us) ts_tablet_manager.cc:1120] Deleting tablet ca394b7f53ce445e8778dd25d0cf25c7
0506 19:56:40.351531 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.351720 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3634506}
I20240506 19:56:40.356876 13243 tablet_replica.cc:330] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:40.358050 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ca394b7f53ce445e8778dd25d0cf25c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca394b7f53ce445e8778dd25d0cf25c7 already in progress: creating tablet
I20240506 19:56:40.361882 12813 tablet_service.cc:1467] Processing CreateTablet for tablet e1b011e94e2b47d7a51d6816576ea124 (DEFAULT_TABLE table=default.Table_4af9a240535b494fbe65842e4ddc343b [id=e8385d8ad6714acf91d8cb6b0de6469c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.367470 13484 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" instance_seqno: 1715025390684788) as {username='slave'} at 127.12.48.193:33595; Asking this server to re-register.
I20240506 19:56:40.370095 13243 raft_consensus.cc:2229] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:40.370659 13243 raft_consensus.cc:2258] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.347640 13353 raft_consensus.cc:504] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.381922 13353 leader_election.cc:288] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.386619 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b011e94e2b47d7a51d6816576ea124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.386704 13629 raft_consensus.cc:504] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.386855 13484 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" instance_seqno: 1715025391180391) as {username='slave'} at 127.12.48.194:49541; Asking this server to re-register.
I20240506 19:56:40.387691 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20aa06290b9a4814a006c423ccd8fc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.388124 13243 ts_tablet_manager.cc:1875] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.388500 13249 tablet_bootstrap.cc:654] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.389132 13035 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:40.389722 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373e04e93a5543f9934077006eaf2ba9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:40.393083 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 67130.1us
I20240506 19:56:40.397820 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ef5a2799b447f992207452dbe39645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.399706 13719 raft_consensus.cc:482] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.399853 13719 raft_consensus.cc:504] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.400342 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b011e94e2b47d7a51d6816576ea124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.401068 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1750c4dd8cdc40509c553b63d166c466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.403085 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458314e4653748e48cf646c72373fc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.404964 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43da1d12ce647dcbcd31263932b2f54. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:40.404904 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 132698us
I20240506 19:56:40.409500 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f759b560e2ab41a6a076976ccca811d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.412101 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66f1e46cce47639fe644fb4ed2e5c0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:40.414705 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d53f3daaa2949a2b6883435f031dc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.414778 13484 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" instance_seqno: 1715025391726585) as {username='slave'} at 127.12.48.195:35545; Asking this server to re-register.
I20240506 19:56:40.415846 13171 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:40.417179 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddae4b0d8d0943d99f54904266f293d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.421567 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da091c44824bd6bb8d390a5fb01ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.426155 13090 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 366) took 1030 ms. Trace:
I20240506 19:56:40.426672 13484 ts_manager.cc:194] Registered new tserver with Master: 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.426365 13090 rpcz_store.cc:276] 0506 19:56:39.395240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.395287 (+ 47us) service_pool.cc:225] Handling call
0506 19:56:39.661428 (+266141us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.661455 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.385846 (+724391us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.425143 (+ 39297us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.425321 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.427026 13101 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 371) took 1024 ms. Trace:
I20240506 19:56:39.629603 13505 rpcz_store.cc:276] 0506 19:56:37.427161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.427239 (+ 78us) service_pool.cc:225] Handling call
0506 19:56:38.393069 (+965830us) raft_consensus.cc:1457] Updating replica for 309 ops
0506 19:56:38.493213 (+100144us) raft_consensus.cc:1520] Early marking committed up to index 19
0506 19:56:38.732578 (+239365us) raft_consensus.cc:1525] Triggering prepare for 309 ops
0506 19:56:38.733017 (+ 439us) write_op.cc:270] Start()
0506 19:56:38.733081 (+ 64us) write_op.cc:276] Timestamp: P: 1715025391940209 usec, L: 0
0506 19:56:38.733431 (+ 350us) write_op.cc:270] Start()
0506 19:56:38.733461 (+ 30us) write_op.cc:276] Timestamp: P: 1715025391948860 usec, L: 0
0506 19:56:38.733643 (+ 182us) write_op.cc:270] Start()
0506 19:56:38.733685 (+ 42us) write_op.cc:276] Timestamp: P: 1715025391950393 usec, L: 0
0506 19:56:38.733842 (+ 157us) write_op.cc:270] Start()
0506 19:56:38.733869 (+ 27us) write_op.cc:276] Timestamp: P: 1715025391951577 usec, L: 0
0506 19:56:38.734047 (+ 178us) write_op.cc:270] Start()
0506 19:56:38.734066 (+ 19us) write_op.cc:276] Timestamp: P: 1715025391952832 usec, L: 0
0506 19:56:38.734181 (+ 115us) write_op.cc:270] Start()
0506 19:56:38.734199 (+ 18us) write_op.cc:276] Timestamp: P: 1715025391954005 usec, L: 0
0506 19:56:38.734329 (+ 130us) write_op.cc:270] Start()
0506 19:56:38.734377 (+ 48us) write_op.cc:276] Timestamp: P: 1715025392094392 usec, L: 0
0506 19:56:38.734521 (+ 144us) write_op.cc:270] Start()
0506 19:56:38.734540 (+ 19us) write_op.cc:276] Timestamp: P: 1715025392107046 usec, L: 0
0506 19:56:38.734703 (+ 163us) write_op.cc:270] Start()
0506 19:56:38.734721 (+ 18us) write_op.cc:276] Timestamp: P: 1715025392119178 usec, L: 0
0506 19:56:38.734895 (+ 174us) write_op.cc:270] Start()
0506 19:56:38.734913 (+ 18us) write_op.cc:276] Timestamp: P: 1715025392119846 usec, L: 0
0506 19:56:38.735040 (+ 127us) write_op.cc:270] Start()
0506 19:56:38.735057 (+ 17us) write_op.cc:276] Timestamp: P: 1715025392120385 usec, L: 0
0506 19:56:38.735179 (+ 122us) write_op.cc:270] Start()
0506 19:56:38.735198 (+ 19us) write_op.cc:276] Timestamp: P: 1715025392120835 usec, L: 0
0506 19:56:38.735322 (+ 124us) write_op.cc:270] Start()
0506 19:56:38.735343 (+ 21us) write_op.cc:276] Timestamp: P: 1715025392143264 usec, L: 0
0506 19:56:38.735472 (+ 129us) write_op.cc:270] Start()
0506 19:56:38.735493 (+ 21us) write_op.cc:276] Timestamp: P: 1715025392144020 usec, L: 0
0506 19:56:38.735690 (+ 197us) write_op.cc:270] Start()
0506 19:56:38.943445 (+207755us) write_op.cc:276] Timestamp: P: 1715025392144554 usec, L: 0
0506 19:56:38.944544 (+ 1099us) write_op.cc:270] Start()
0506 19:56:38.944672 (+ 128us) write_op.cc:276] Timestamp: P: 1715025392153963 usec, L: 0
0506 19:56:38.945496 (+ 824us) write_op.cc:270] Start()
0506 19:56:38.945747 (+ 251us) write_op.cc:276] Timestamp: P: 1715025392158012 usec, L: 0
0506 19:56:38.950239 (+ 4492us) write_op.cc:270] Start()
0506 19:56:38.950376 (+ 137us) write_op.cc:276] Timestamp: P: 1715025392159100 usec, L: 0
0506 19:56:38.951187 (+ 811us) write_op.cc:270] Start()
0506 19:56:38.951311 (+ 124us) write_op.cc:276] Timestamp: P: 1715025392159981 usec, L: 0
0506 19:56:38.952118 (+ 807us) write_op.cc:270] Start()
0506 19:56:38.952314 (+ 196us) write_op.cc:276] Timestamp: P: 1715025392160816 usec, L: 0
0506 19:56:38.964508 (+ 12194us) write_op.cc:270] Start()
0506 19:56:38.964747 (+ 239us) write_op.cc:276] Timestamp: P: 1715025392161590 usec, L: 0
0506 19:56:38.966942 (+ 2195us) write_op.cc:270] Start()
0506 19:56:38.967147 (+ 205us) write_op.cc:276] Timestamp: P: 1715025392164252 usec, L: 0
0506 19:56:38.968364 (+ 1217us) write_op.cc:270] Start()
0506 19:56:38.968395 (+ 31us) write_op.cc:276] Timestamp: P: 1715025392217738 usec, L: 0
0506 19:56:38.970778 (+ 2383us) write_op.cc:270] Start()
0506 19:56:38.971047 (+ 269us) write_op.cc:276] Timestamp: P: 1715025392218836 usec, L: 0
0506 19:56:38.972454 (+ 1407us) write_op.cc:270] Start()
0506 19:56:38.972653 (+ 199us) write_op.cc:276] Timestamp: P: 1715025392219632 usec, L: 0
0506 19:56:38.973670 (+ 1017us) write_op.cc:270] Start()
0506 19:56:38.973865 (+ 195us) write_op.cc:276] Timestamp: P: 1715025392237403 usec, L: 0
0506 19:56:38.975055 (+ 1190us) write_op.cc:270] Start()
0506 19:56:38.975246 (+ 191us) write_op.cc:276] Timestamp: P: 1715025392267377 usec, L: 0
0506 19:56:38.976931 (+ 1685us) write_op.cc:270] Start()
0506 19:56:38.977126 (+ 195us) write_op.cc:276] Timestamp: P: 1715025392269064 usec, L: 0
0506 19:56:39.031678 (+ 54552us) write_op.cc:270] Start()
0506 19:56:39.031720 (+ 42us) write_op.cc:276] Timestamp: P: 1715025392310092 usec, L: 0
0506 19:56:39.034810 (+ 3090us) write_op.cc:270] Start()
0506 19:56:39.034859 (+ 49us) write_op.cc:276] Timestamp: P: 1715025392361099 usec, L: 0
0506 19:56:39.035207 (+ 348us) write_op.cc:270] Start()
0506 19:56:39.035235 (+ 28us) write_op.cc:276] Timestamp: P: 1715025392395268 usec, L: 0
0506 19:56:39.036938 (+ 1703us) write_op.cc:270] Start()
0506 19:56:39.036970 (+ 32us) write_op.cc:276] Timestamp: P: 1715025392403333 usec, L: 0
0506 19:56:39.037175 (+ 205us) write_op.cc:270] Start()
0506 19:56:39.037198 (+ 23us) write_op.cc:276] Timestamp: P: 1715025392403943 usec, L: 0
0506 19:56:39.037357 (+ 159us) write_op.cc:270] Start()
0506 19:56:39.037382 (+ 25us) write_op.cc:276] Timestamp: P: 1715025392404633 usec, L: 0
0506 19:56:39.037514 (+ 132us) write_op.cc:270] Start()
0506 19:56:39.037537 (+ 23us) write_op.cc:276] Timestamp: P: 1715025392406038 usec, L: 0
0506 19:56:39.037673 (+ 136us) write_op.cc:270] Start()
0506 19:56:39.037695 (+ 22us) write_op.cc:276] Timestamp: P: 1715025392408392 usec, L: 0
0506 19:56:39.037811 (+ 116us) write_op.cc:270] Start()
0506 19:56:39.037891 (+ 80us) write_op.cc:276] Timestamp: P: 1715025392445345 usec, L: 0
0506 19:56:39.038055 (+ 164us) write_op.cc:270] Start()
0506 19:56:39.038099 (+ 44us) write_op.cc:276] Timestamp: P: 1715025392526504 usec, L: 0
0506 19:56:39.038257 (+ 158us) write_op.cc:270] Start()
0506 19:56:39.038277 (+ 20us) write_op.cc:276] Timestamp: P: 1715025392539926 usec, L: 0
0506 19:56:39.039405 (+ 1128us) write_op.cc:270] Start()
0506 19:56:39.043332 (+ 3927us) write_op.cc:276] Timestamp: P: 1715025392540676 usec, L: 0
0506 19:56:39.043609 (+ 277us) write_op.cc:270] Start()
0506 19:56:39.043640 (+ 31us) write_op.cc:276] Timestamp: P: 1715025392541303 usec, L: 0
0506 19:56:39.043816 (+ 176us) write_op.cc:270] Start()
0506 19:56:39.043841 (+ 25us) write_op.cc:276] Timestamp: P: 1715025392541984 usec, L: 0
0506 19:56:39.043999 (+ 158us) write_op.cc:270] Start()
0506 19:56:39.044024 (+ 25us) write_op.cc:276] Timestamp: P: 1715025392545847 usec, L: 0
0506 19:56:39.044162 (+ 138us) write_op.cc:270] Start()
0506 19:56:39.044181 (+ 19us) write_op.cc:276] Timestamp: P: 1715025392550740 usec, L: 0
0506 19:56:39.044307 (+ 126us) write_op.cc:270] Start()
0506 19:56:39.044331 (+ 24us) write_op.cc:276] Timestamp: P: 1715025392574083 usec, L: 0
0506 19:56:39.044466 (+ 135us) write_op.cc:270] Start()
0506 19:56:39.044488 (+ 22us) write_op.cc:276] Timestamp: P: 1715025392575090 usec, L: 0
0506 19:56:39.044622 (+ 134us) write_op.cc:270] Start()
0506 19:56:39.044643 (+ 21us) write_op.cc:276] Timestamp: P: 1715025392575768 usec, L: 0
0506 19:56:39.044789 (+ 146us) write_op.cc:270] Start()
0506 19:56:39.044812 (+ 23us) write_op.cc:276] Timestamp: P: 1715025392576471 usec, L: 0
0506 19:56:39.045063 (+ 251us) write_op.cc:270] Start()
0506 19:56:39.045085 (+ 22us) write_op.cc:276] Timestamp: P: 1715025392577080 usec, L: 0
0506 19:56:39.045225 (+ 140us) write_op.cc:270] Start()
0506 19:56:39.045242 (+ 17us) write_op.cc:276] Timestamp: P: 1715025392607236 usec, L: 0
0506 19:56:39.045405 (+ 163us) write_op.cc:270] Start()
0506 19:56:39.045429 (+ 24us) write_op.cc:276] Timestamp: P: 1715025392677804 usec, L: 0
0506 19:56:39.045615 (+ 186us) write_op.cc:270] Start()
0506 19:56:39.045634 (+ 19us) write_op.cc:276] Timestamp: P: 1715025392690504 usec, L: 0
0506 19:56:39.045754 (+ 120us) write_op.cc:270] Start()
0506 19:56:39.045770 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392694536 usec, L: 0
0506 19:56:39.045883 (+ 113us) write_op.cc:270] Start()
0506 19:56:39.045925 (+ 42us) write_op.cc:276] Timestamp: P: 1715025392698724 usec, L: 0
0506 19:56:39.046056 (+ 131us) write_op.cc:270] Start()
0506 19:56:39.046075 (+ 19us) write_op.cc:276] Timestamp: P: 1715025392701723 usec, L: 0
0506 19:56:39.046209 (+ 134us) write_op.cc:270] Start()
0506 19:56:39.046228 (+ 19us) write_op.cc:276] Timestamp: P: 1715025392702487 usec, L: 0
0506 19:56:39.046366 (+ 138us) write_op.cc:270] Start()
0506 19:56:39.046388 (+ 22us) write_op.cc:276] Timestamp: P: 1715025392786686 usec, L: 0
0506 19:56:39.046501 (+ 113us) write_op.cc:270] Start()
0506 19:56:39.046521 (+ 20us) write_op.cc:276] Timestamp: P: 1715025392795696 usec, L: 0
0506 19:56:39.047603 (+ 1082us) write_op.cc:270] Start()
0506 19:56:39.047637 (+ 34us) write_op.cc:276] Timestamp: P: 1715025392798935 usec, L: 0
0506 19:56:39.054688 (+ 7051us) write_op.cc:270] Start()
0506 19:56:39.054750 (+ 62us) write_op.cc:276] Timestamp: P: 1715025392811002 usec, L: 0
0506 19:56:39.054962 (+ 212us) write_op.cc:270] Start()
0506 19:56:39.054979 (+ 17us) write_op.cc:276] Timestamp: P: 1715025392812568 usec, L: 0
0506 19:56:39.055098 (+ 119us) write_op.cc:270] Start()
0506 19:56:39.055113 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392814015 usec, L: 0
0506 19:56:39.055221 (+ 108us) write_op.cc:270] Start()
0506 19:56:39.055236 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392818726 usec, L: 0
0506 19:56:39.055340 (+ 104us) write_op.cc:270] Start()
0506 19:56:39.055356 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392820585 usec, L: 0
0506 19:56:39.055492 (+ 136us) write_op.cc:270] Start()
0506 19:56:39.055509 (+ 17us) write_op.cc:276] Timestamp: P: 1715025392832711 usec, L: 0
0506 19:56:39.055629 (+ 120us) write_op.cc:270] Start()
0506 19:56:39.055645 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392833818 usec, L: 0
0506 19:56:39.055741 (+ 96us) write_op.cc:270] Start()
0506 19:56:39.055756 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392834788 usec, L: 0
0506 19:56:39.055852 (+ 96us) write_op.cc:270] Start()
0506 19:56:39.055866 (+ 14us) write_op.cc:276] Timestamp: P: 1715025392835705 usec, L: 0
0506 19:56:39.055961 (+ 95us) write_op.cc:270] Start()
0506 19:56:39.055975 (+ 14us) write_op.cc:276] Timestamp: P: 1715025392836573 usec, L: 0
0506 19:56:39.056067 (+ 92us) write_op.cc:270] Start()
0506 19:56:39.056098 (+ 31us) write_op.cc:276] Timestamp: P: 1715025392837435 usec, L: 0
0506 19:56:39.056203 (+ 105us) write_op.cc:270] Start()
0506 19:56:39.056218 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392838378 usec, L: 0
0506 19:56:39.056342 (+ 124us) write_op.cc:270] Start()
0506 19:56:39.056358 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392847956 usec, L: 0
0506 19:56:39.056480 (+ 122us) write_op.cc:270] Start()
0506 19:56:39.056496 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392849139 usec, L: 0
0506 19:56:39.056600 (+ 104us) write_op.cc:270] Start()
0506 19:56:39.056615 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392850214 usec, L: 0
0506 19:56:39.056735 (+ 120us) write_op.cc:270] Start()
0506 19:56:39.056750 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392857253 usec, L: 0
0506 19:56:39.056855 (+ 105us) write_op.cc:270] Start()
0506 19:56:39.056871 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392870942 usec, L: 0
0506 19:56:39.056975 (+ 104us) write_op.cc:270] Start()
0506 19:56:39.056991 (+ 16us) write_op.cc:276] Timestamp: P: 1715025392908808 usec, L: 0
0506 19:56:39.057083 (+ 92us) write_op.cc:270] Start()
0506 19:56:39.057098 (+ 15us) write_op.cc:276] Timestamp: P: 1715025392913869 usec, L: 0
0506 19:56:39.057193 (+ 95us) write_op.cc:270] Start()
0506 19:56:39.057228 (+ 35us) write_op.cc:276] Timestamp: P: 1715025392924013 usec, L: 0
0506 19:56:39.057396 (+ 168us) write_op.cc:270] Start()
0506 19:56:39.057417 (+ 21us) write_op.cc:276] Timestamp: P: 1715025392937534 usec, L: 0
0506 19:56:39.057582 (+ 165us) write_op.cc:270] Start()
0506 19:56:39.057605 (+ 23us) write_op.cc:276] Timestamp: P: 1715025392938894 usec, L: 0
0506 19:56:39.057741 (+ 136us) write_op.cc:270] Start()
0506 19:56:39.057763 (+ 22us) write_op.cc:276] Timestamp: P: 1715025392945513 usec, L: 0
0506 19:56:39.057895 (+ 132us) write_op.cc:270] Start()
0506 19:56:39.057916 (+ 21us) write_op.cc:276] Timestamp: P: 1715025392957543 usec, L: 0
0506 19:56:39.058069 (+ 153us) write_op.cc:270] Start()
0506 19:56:39.058092 (+ 23us) write_op.cc:276] Timestamp: P: 1715025392960943 usec, L: 0
0506 19:56:39.058234 (+ 142us) write_op.cc:270] Start()
0506 19:56:39.058256 (+ 22us) write_op.cc:276] Timestamp: P: 1715025392986087 usec, L: 0
0506 19:56:39.058386 (+ 130us) write_op.cc:270] Start()
0506 19:56:39.058413 (+ 27us) write_op.cc:276] Timestamp: P: 1715025393013600 usec, L: 0
0506 19:56:39.069531 (+ 11118us) write_op.cc:270] Start()
0506 19:56:39.069573 (+ 42us) write_op.cc:276] Timestamp: P: 1715025393279299 usec, L: 0
0506 19:56:39.069764 (+ 191us) write_op.cc:270] Start()
0506 19:56:39.069781 (+ 17us) write_op.cc:276] Timestamp: P: 1715025393295769 usec, L: 0
0506 19:56:39.069925 (+ 144us) write_op.cc:270] Start()
0506 19:56:39.069955 (+ 30us) write_op.cc:276] Timestamp: P: 1715025393316442 usec, L: 0
0506 19:56:39.070106 (+ 151us) write_op.cc:270] Start()
0506 19:56:39.070124 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393344863 usec, L: 0
0506 19:56:39.070286 (+ 162us) write_op.cc:270] Start()
0506 19:56:39.090336 (+ 20050us) write_op.cc:276] Timestamp: P: 1715025393346569 usec, L: 0
0506 19:56:39.090663 (+ 327us) write_op.cc:270] Start()
0506 19:56:39.090705 (+ 42us) write_op.cc:276] Timestamp: P: 1715025393347364 usec, L: 0
0506 19:56:39.090839 (+ 134us) write_op.cc:270] Start()
0506 19:56:39.090857 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393375115 usec, L: 0
0506 19:56:39.091050 (+ 193us) write_op.cc:270] Start()
0506 19:56:39.091070 (+ 20us) write_op.cc:276] Timestamp: P: 1715025393376514 usec, L: 0
0506 19:56:39.091186 (+ 116us) write_op.cc:270] Start()
0506 19:56:39.091203 (+ 17us) write_op.cc:276] Timestamp: P: 1715025393377429 usec, L: 0
0506 19:56:39.091343 (+ 140us) write_op.cc:270] Start()
0506 19:56:39.091361 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393378250 usec, L: 0
0506 19:56:39.091477 (+ 116us) write_op.cc:270] Start()
0506 19:56:39.091497 (+ 20us) write_op.cc:276] Timestamp: P: 1715025393379046 usec, L: 0
0506 19:56:39.091602 (+ 105us) write_op.cc:270] Start()
0506 19:56:39.091618 (+ 16us) write_op.cc:276] Timestamp: P: 1715025393379843 usec, L: 0
0506 19:56:39.091751 (+ 133us) write_op.cc:270] Start()
0506 19:56:39.091771 (+ 20us) write_op.cc:276] Timestamp: P: 1715025393405919 usec, L: 0
0506 19:56:39.091923 (+ 152us) write_op.cc:270] Start()
0506 19:56:39.091999 (+ 76us) write_op.cc:276] Timestamp: P: 1715025393432509 usec, L: 0
0506 19:56:39.092126 (+ 127us) write_op.cc:270] Start()
0506 19:56:39.092144 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393454498 usec, L: 0
0506 19:56:39.092260 (+ 116us) write_op.cc:270] Start()
0506 19:56:39.092297 (+ 37us) write_op.cc:276] Timestamp: P: 1715025393455604 usec, L: 0
0506 19:56:39.092413 (+ 116us) write_op.cc:270] Start()
0506 19:56:39.092435 (+ 22us) write_op.cc:276] Timestamp: P: 1715025393456997 usec, L: 0
0506 19:56:39.092922 (+ 487us) write_op.cc:270] Start()
0506 19:56:39.092951 (+ 29us) write_op.cc:276] Timestamp: P: 1715025393457978 usec, L: 0
0506 19:56:39.093084 (+ 133us) write_op.cc:270] Start()
0506 19:56:39.093102 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393458800 usec, L: 0
0506 19:56:39.093207 (+ 105us) write_op.cc:270] Start()
0506 19:56:39.093225 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393462838 usec, L: 0
0506 19:56:39.093380 (+ 155us) write_op.cc:270] Start()
0506 19:56:39.093414 (+ 34us) write_op.cc:276] Timestamp: P: 1715025393473937 usec, L: 0
0506 19:56:39.097589 (+ 4175us) write_op.cc:270] Start()
0506 19:56:39.097623 (+ 34us) write_op.cc:276] Timestamp: P: 1715025393499330 usec, L: 0
0506 19:56:39.097772 (+ 149us) write_op.cc:270] Start()
0506 19:56:39.097797 (+ 25us) write_op.cc:276] Timestamp: P: 1715025393500046 usec, L: 0
0506 19:56:39.097931 (+ 134us) write_op.cc:270] Start()
0506 19:56:39.097959 (+ 28us) write_op.cc:276] Timestamp: P: 1715025393502116 usec, L: 0
0506 19:56:39.098062 (+ 103us) write_op.cc:270] Start()
0506 19:56:39.098079 (+ 17us) write_op.cc:276] Timestamp: P: 1715025393502969 usec, L: 0
0506 19:56:39.098189 (+ 110us) write_op.cc:270] Start()
0506 19:56:39.098204 (+ 15us) write_op.cc:276] Timestamp: P: 1715025393503515 usec, L: 0
0506 19:56:39.098331 (+ 127us) write_op.cc:270] Start()
0506 19:56:39.098347 (+ 16us) write_op.cc:276] Timestamp: P: 1715025393560471 usec, L: 0
0506 19:56:39.098457 (+ 110us) write_op.cc:270] Start()
0506 19:56:39.098475 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393586980 usec, L: 0
0506 19:56:39.098602 (+ 127us) write_op.cc:270] Start()
0506 19:56:39.098619 (+ 17us) write_op.cc:276] Timestamp: P: 1715025393634898 usec, L: 0
0506 19:56:39.098750 (+ 131us) write_op.cc:270] Start()
0506 19:56:39.098768 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393658884 usec, L: 0
0506 19:56:39.098907 (+ 139us) write_op.cc:270] Start()
0506 19:56:39.098931 (+ 24us) write_op.cc:276] Timestamp: P: 1715025393659974 usec, L: 0
0506 19:56:39.099070 (+ 139us) write_op.cc:270] Start()
0506 19:56:39.099100 (+ 30us) write_op.cc:276] Timestamp: P: 1715025393660672 usec, L: 0
0506 19:56:39.099242 (+ 142us) write_op.cc:270] Start()
0506 19:56:39.099264 (+ 22us) write_op.cc:276] Timestamp: P: 1715025393675577 usec, L: 0
0506 19:56:39.099383 (+ 119us) write_op.cc:270] Start()
0506 19:56:39.099404 (+ 21us) write_op.cc:276] Timestamp: P: 1715025393676459 usec, L: 0
0506 19:56:39.106879 (+ 7475us) write_op.cc:270] Start()
0506 19:56:39.106928 (+ 49us) write_op.cc:276] Timestamp: P: 1715025393677183 usec, L: 0
0506 19:56:39.107160 (+ 232us) write_op.cc:270] Start()
0506 19:56:39.107185 (+ 25us) write_op.cc:276] Timestamp: P: 1715025393677796 usec, L: 0
0506 19:56:39.107314 (+ 129us) write_op.cc:270] Start()
0506 19:56:39.107333 (+ 19us) write_op.cc:276] Timestamp: P: 1715025393678424 usec, L: 0
0506 19:56:39.107444 (+ 111us) write_op.cc:270] Start()
0506 19:56:39.107465 (+ 21us) write_op.cc:276] Timestamp: P: 1715025393711928 usec, L: 0
0506 19:56:39.107623 (+ 158us) write_op.cc:270] Start()
0506 19:56:39.107642 (+ 19us) write_op.cc:276] Timestamp: P: 1715025393713444 usec, L: 0
0506 19:56:39.107796 (+ 154us) write_op.cc:270] Start()
0506 19:56:39.107814 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393730414 usec, L: 0
0506 19:56:39.107954 (+ 140us) write_op.cc:270] Start()
0506 19:56:39.107973 (+ 19us) write_op.cc:276] Timestamp: P: 1715025393734381 usec, L: 0
0506 19:56:39.108133 (+ 160us) write_op.cc:270] Start()
0506 19:56:39.108153 (+ 20us) write_op.cc:276] Timestamp: P: 1715025393738598 usec, L: 0
0506 19:56:39.108304 (+ 151us) write_op.cc:270] Start()
0506 19:56:39.108323 (+ 19us) write_op.cc:276] Timestamp: P: 1715025393791998 usec, L: 0
0506 19:56:39.108436 (+ 113us) write_op.cc:270] Start()
0506 19:56:39.108453 (+ 17us) write_op.cc:276] Timestamp: P: 1715025393821189 usec, L: 0
0506 19:56:39.108566 (+ 113us) write_op.cc:270] Start()
0506 19:56:39.108623 (+ 57us) write_op.cc:276] Timestamp: P: 1715025393828444 usec, L: 0
0506 19:56:39.108753 (+ 130us) write_op.cc:270] Start()
0506 19:56:39.108772 (+ 19us) write_op.cc:276] Timestamp: P: 1715025393846256 usec, L: 0
0506 19:56:39.108906 (+ 134us) write_op.cc:270] Start()
0506 19:56:39.108925 (+ 19us) write_op.cc:276] Timestamp: P: 1715025393867333 usec, L: 0
0506 19:56:39.115726 (+ 6801us) write_op.cc:270] Start()
0506 19:56:39.115778 (+ 52us) write_op.cc:276] Timestamp: P: 1715025393898013 usec, L: 0
0506 19:56:39.116029 (+ 251us) write_op.cc:270] Start()
0506 19:56:39.116053 (+ 24us) write_op.cc:276] Timestamp: P: 1715025393898726 usec, L: 0
0506 19:56:39.116189 (+ 136us) write_op.cc:270] Start()
0506 19:56:39.116205 (+ 16us) write_op.cc:276] Timestamp: P: 1715025393899317 usec, L: 0
0506 19:56:39.116310 (+ 105us) write_op.cc:270] Start()
0506 19:56:39.116325 (+ 15us) write_op.cc:276] Timestamp: P: 1715025393917871 usec, L: 0
0506 19:56:39.116412 (+ 87us) write_op.cc:270] Start()
0506 19:56:39.116426 (+ 14us) write_op.cc:276] Timestamp: P: 1715025393951083 usec, L: 0
0506 19:56:39.116551 (+ 125us) write_op.cc:270] Start()
0506 19:56:39.116571 (+ 20us) write_op.cc:276] Timestamp: P: 1715025393952828 usec, L: 0
0506 19:56:39.116692 (+ 121us) write_op.cc:270] Start()
0506 19:56:39.116715 (+ 23us) write_op.cc:276] Timestamp: P: 1715025393953892 usec, L: 0
0506 19:56:39.116831 (+ 116us) write_op.cc:270] Start()
0506 19:56:39.116847 (+ 16us) write_op.cc:276] Timestamp: P: 1715025393956294 usec, L: 0
0506 19:56:39.116976 (+ 129us) write_op.cc:270] Start()
0506 19:56:39.116994 (+ 18us) write_op.cc:276] Timestamp: P: 1715025393959594 usec, L: 0
0506 19:56:39.117100 (+ 106us) write_op.cc:270] Start()
0506 19:56:39.117119 (+ 19us) write_op.cc:276] Timestamp: P: 1715025394015380 usec, L: 0
0506 19:56:39.117248 (+ 129us) write_op.cc:270] Start()
0506 19:56:39.117266 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394073110 usec, L: 0
0506 19:56:39.117403 (+ 137us) write_op.cc:270] Start()
0506 19:56:39.117424 (+ 21us) write_op.cc:276] Timestamp: P: 1715025394081424 usec, L: 0
0506 19:56:39.117541 (+ 117us) write_op.cc:270] Start()
0506 19:56:39.117560 (+ 19us) write_op.cc:276] Timestamp: P: 1715025394082254 usec, L: 0
0506 19:56:39.117672 (+ 112us) write_op.cc:270] Start()
0506 19:56:39.117692 (+ 20us) write_op.cc:276] Timestamp: P: 1715025394082870 usec, L: 0
0506 19:56:39.117832 (+ 140us) write_op.cc:270] Start()
0506 19:56:39.117852 (+ 20us) write_op.cc:276] Timestamp: P: 1715025394092303 usec, L: 0
0506 19:56:39.118009 (+ 157us) write_op.cc:270] Start()
0506 19:56:39.118027 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394092997 usec, L: 0
0506 19:56:39.118298 (+ 271us) write_op.cc:270] Start()
0506 19:56:39.118322 (+ 24us) write_op.cc:276] Timestamp: P: 1715025394097865 usec, L: 0
0506 19:56:39.118488 (+ 166us) write_op.cc:270] Start()
0506 19:56:39.118512 (+ 24us) write_op.cc:276] Timestamp: P: 1715025394099566 usec, L: 0
0506 19:56:39.118663 (+ 151us) write_op.cc:270] Start()
0506 19:56:39.118684 (+ 21us) write_op.cc:276] Timestamp: P: 1715025394118925 usec, L: 0
0506 19:56:39.118969 (+ 285us) write_op.cc:270] Start()
0506 19:56:39.118995 (+ 26us) write_op.cc:276] Timestamp: P: 1715025394124969 usec, L: 0
0506 19:56:39.119158 (+ 163us) write_op.cc:270] Start()
0506 19:56:39.119181 (+ 23us) write_op.cc:276] Timestamp: P: 1715025394125637 usec, L: 0
0506 19:56:39.119294 (+ 113us) write_op.cc:270] Start()
0506 19:56:39.119314 (+ 20us) write_op.cc:276] Timestamp: P: 1715025394152362 usec, L: 0
0506 19:56:39.119436 (+ 122us) write_op.cc:270] Start()
0506 19:56:39.119455 (+ 19us) write_op.cc:276] Timestamp: P: 1715025394164623 usec, L: 0
0506 19:56:39.119580 (+ 125us) write_op.cc:270] Start()
0506 19:56:39.119597 (+ 17us) write_op.cc:276] Timestamp: P: 1715025394170201 usec, L: 0
0506 19:56:39.119720 (+ 123us) write_op.cc:270] Start()
0506 19:56:39.119746 (+ 26us) write_op.cc:276] Timestamp: P: 1715025394171303 usec, L: 0
0506 19:56:39.119904 (+ 158us) write_op.cc:270] Start()
0506 19:56:39.119923 (+ 19us) write_op.cc:276] Timestamp: P: 1715025394245119 usec, L: 0
0506 19:56:39.120067 (+ 144us) write_op.cc:270] Start()
0506 19:56:39.120091 (+ 24us) write_op.cc:276] Timestamp: P: 1715025394269886 usec, L: 0
0506 19:56:39.120228 (+ 137us) write_op.cc:270] Start()
0506 19:56:39.120246 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394270897 usec, L: 0
0506 19:56:39.120377 (+ 131us) write_op.cc:270] Start()
0506 19:56:39.120395 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394282527 usec, L: 0
0506 19:56:39.120525 (+ 130us) write_op.cc:270] Start()
0506 19:56:39.120543 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394283953 usec, L: 0
0506 19:56:39.120666 (+ 123us) write_op.cc:270] Start()
0506 19:56:39.120684 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394330144 usec, L: 0
0506 19:56:39.120807 (+ 123us) write_op.cc:270] Start()
0506 19:56:39.120829 (+ 22us) write_op.cc:276] Timestamp: P: 1715025394363402 usec, L: 0
0506 19:56:39.120976 (+ 147us) write_op.cc:270] Start()
0506 19:56:39.121000 (+ 24us) write_op.cc:276] Timestamp: P: 1715025394365513 usec, L: 0
0506 19:56:39.121179 (+ 179us) write_op.cc:270] Start()
0506 19:56:39.126971 (+ 5792us) write_op.cc:276] Timestamp: P: 1715025394367649 usec, L: 0
0506 19:56:39.127231 (+ 260us) write_op.cc:270] Start()
0506 19:56:39.127256 (+ 25us) write_op.cc:276] Timestamp: P: 1715025394368213 usec, L: 0
0506 19:56:39.127373 (+ 117us) write_op.cc:270] Start()
0506 19:56:39.127390 (+ 17us) write_op.cc:276] Timestamp: P: 1715025394375502 usec, L: 0
0506 19:56:39.127531 (+ 141us) write_op.cc:270] Start()
0506 19:56:39.127555 (+ 24us) write_op.cc:276] Timestamp: P: 1715025394376218 usec, L: 0
0506 19:56:39.127687 (+ 132us) write_op.cc:270] Start()
0506 19:56:39.127709 (+ 22us) write_op.cc:276] Timestamp: P: 1715025394376799 usec, L: 0
0506 19:56:39.127840 (+ 131us) write_op.cc:270] Start()
0506 19:56:39.127860 (+ 20us) write_op.cc:276] Timestamp: P: 1715025394377340 usec, L: 0
0506 19:56:39.128027 (+ 167us) write_op.cc:270] Start()
0506 19:56:39.128050 (+ 23us) write_op.cc:276] Timestamp: P: 1715025394387371 usec, L: 0
0506 19:56:39.128663 (+ 613us) write_op.cc:270] Start()
0506 19:56:39.128696 (+ 33us) write_op.cc:276] Timestamp: P: 1715025394441314 usec, L: 0
0506 19:56:39.128981 (+ 285us) write_op.cc:270] Start()
0506 19:56:39.129009 (+ 28us) write_op.cc:276] Timestamp: P: 1715025394445639 usec, L: 0
0506 19:56:39.129172 (+ 163us) write_op.cc:270] Start()
0506 19:56:39.129195 (+ 23us) write_op.cc:276] Timestamp: P: 1715025394461510 usec, L: 0
0506 19:56:39.129430 (+ 235us) write_op.cc:270] Start()
0506 19:56:39.129457 (+ 27us) write_op.cc:276] Timestamp: P: 1715025394464602 usec, L: 0
0506 19:56:39.129723 (+ 266us) write_op.cc:270] Start()
0506 19:56:39.129750 (+ 27us) write_op.cc:276] Timestamp: P: 1715025394495423 usec, L: 0
0506 19:56:39.129935 (+ 185us) write_op.cc:270] Start()
0506 19:56:39.129969 (+ 34us) write_op.cc:276] Timestamp: P: 1715025394523371 usec, L: 0
0506 19:56:39.130127 (+ 158us) write_op.cc:270] Start()
0506 19:56:39.130149 (+ 22us) write_op.cc:276] Timestamp: P: 1715025394541827 usec, L: 0
0506 19:56:39.130308 (+ 159us) write_op.cc:270] Start()
0506 19:56:39.130333 (+ 25us) write_op.cc:276] Timestamp: P: 1715025394546888 usec, L: 0
0506 19:56:39.130541 (+ 208us) write_op.cc:270] Start()
0506 19:56:39.130568 (+ 27us) write_op.cc:276] Timestamp: P: 1715025394580049 usec, L: 0
0506 19:56:39.130811 (+ 243us) write_op.cc:270] Start()
0506 19:56:39.130829 (+ 18us) write_op.cc:276] Timestamp: P: 1715025394589751 usec, L: 0
0506 19:56:39.130967 (+ 138us) write_op.cc:270] Start()
0506 19:56:39.130989 (+ 22us) write_op.cc:276] Timestamp: P: 1715025394590450 usec, L: 0
0506 19:56:39.131131 (+ 142us) write_op.cc:270] Start()
0506 19:56:39.131157 (+ 26us) write_op.cc:276] Timestamp: P: 1715025394590979 usec, L: 0
0506 19:56:39.131320 (+ 163us) write_op.cc:270] Start()
0506 19:56:39.131344 (+ 24us) write_op.cc:276] Timestamp: P: 1715025394598723 usec, L: 0
0506 19:56:39.131458 (+ 114us) write_op.cc:270] Start()
0506 19:56:39.131477 (+ 19us) write_op.cc:276] Timestamp: P: 1715025394616171 usec, L: 0
0506 19:56:39.131609 (+ 132us) write_op.cc:270] Start()
0506 19:56:39.131632 (+ 23us) write_op.cc:276] Timestamp: P: 1715025394695992 usec, L: 0
0506 19:56:39.131780 (+ 148us) write_op.cc:270] Start()
0506 19:56:39.131803 (+ 23us) write_op.cc:276] Timestamp: P: 1715025394980700 usec, L: 0
0506 19:56:39.131947 (+ 144us) write_op.cc:270] Start()
0506 19:56:39.131975 (+ 28us) write_op.cc:276] Timestamp: P: 1715025394981363 usec, L: 0
0506 19:56:39.132132 (+ 157us) write_op.cc:270] Start()
0506 19:56:39.132155 (+ 23us) write_op.cc:276] Timestamp: P: 1715025395016012 usec, L: 0
0506 19:56:39.132300 (+ 145us) write_op.cc:270] Start()
0506 19:56:39.132320 (+ 20us) write_op.cc:276] Timestamp: P: 1715025395016634 usec, L: 0
0506 19:56:39.132455 (+ 135us) write_op.cc:270] Start()
0506 19:56:39.132478 (+ 23us) write_op.cc:276] Timestamp: P: 1715025395017148 usec, L: 0
0506 19:56:39.132638 (+ 160us) write_op.cc:270] Start()
0506 19:56:39.132662 (+ 24us) write_op.cc:276] Timestamp: P: 1715025395017698 usec, L: 0
0506 19:56:39.133203 (+ 541us) write_op.cc:270] Start()
0506 19:56:39.133231 (+ 28us) write_op.cc:276] Timestamp: P: 1715025395018143 usec, L: 0
0506 19:56:39.133430 (+ 199us) write_op.cc:270] Start()
0506 19:56:39.133453 (+ 23us) write_op.cc:276] Timestamp: P: 1715025395018649 usec, L: 0
0506 19:56:39.133590 (+ 137us) write_op.cc:270] Start()
0506 19:56:39.133613 (+ 23us) write_op.cc:276] Timestamp: P: 1715025
I20240506 19:56:40.438232 12821 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 357) took 1009 ms. Trace:
I20240506 19:56:40.438395 12821 rpcz_store.cc:276] 0506 19:56:39.403034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.440225 (+ 37191us) service_pool.cc:225] Handling call
0506 19:56:39.678164 (+237939us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.678188 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.681146 (+ 2958us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.863757 (+182611us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.412716 (+548959us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.438971 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 373e04e93a5543f9934077006eaf2ba9 (DEFAULT_TABLE table=default.Table_5344cd78a3dd4575a939a926d7e1eb16 [id=d6d66e91fd3540d292e12bfeac68617a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.441114 13716 raft_consensus.cc:482] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.441478 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373e04e93a5543f9934077006eaf2ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.441550 12822 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 351) took 1390 ms. Trace:
I20240506 19:56:40.441358 13716 raft_consensus.cc:504] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.442850 12822 rpcz_store.cc:276] 0506 19:56:39.022283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.404458 (+382175us) service_pool.cc:225] Handling call
0506 19:56:39.637478 (+233020us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.637516 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.640676 (+ 3160us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.864088 (+223412us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.412801 (+548713us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.443161 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:40.445003 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddae4b0d8d0943d99f54904266f293d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.446660 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b803ae289344f88b604a2053881d59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.446872 13246 ts_tablet_manager.cc:1888] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:40.447089 12842 raft_consensus.cc:2454] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:40.447553 12816 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 370) took 1006 ms. Trace:
I20240506 19:56:40.443401 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b24fa4663244a88dac9ea99fe38f3a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:40.450230 13246 log.cc:1196] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/6d8a9ecf47d7474c82758417449d940c
I20240506 19:56:40.450904 12810 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 362) took 1045 ms. Trace:
I20240506 19:56:40.451004 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff7050bbc2e47cf823d25ec10b89718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.451179 12816 rpcz_store.cc:276] 0506 19:56:39.440753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.670233 (+229480us) service_pool.cc:225] Handling call
0506 19:56:39.670716 (+ 483us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.670744 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.731829 (+ 61085us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.876012 (+144183us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.447510 (+571498us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9557}
I20240506 19:56:40.451519 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.451777 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd3fc62e9614cf3bde0202ee6832bb3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:40.452128 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 59b022c086d141a19a9b35c4a8c8bef9 (DEFAULT_TABLE table=default.Table_d95065af2929456c8ae9740f3fcffcdc [id=855c174fb8224b5a8acd4c0dd4b4f8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.452277 13099 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 370) took 1052 ms. Trace:
I20240506 19:56:40.452884 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b011e94e2b47d7a51d6816576ea124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.453253 13099 rpcz_store.cc:276] 0506 19:56:39.397553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.397605 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:39.746139 (+348534us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.748710 (+ 2571us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.411302 (+662592us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.449360 (+ 38058us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.449505 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.453465 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b803ae289344f88b604a2053881d59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.453743 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.450258 13101 rpcz_store.cc:276] 0506 19:56:39.398272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.400917 (+ 2645us) service_pool.cc:225] Handling call
0506 19:56:39.745934 (+345017us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.745994 (+ 60us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.410393 (+664399us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.419623 (+ 9230us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.422438 (+ 2815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.454456 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 59b022c086d141a19a9b35c4a8c8bef9 (DEFAULT_TABLE table=default.Table_d95065af2929456c8ae9740f3fcffcdc [id=855c174fb8224b5a8acd4c0dd4b4f8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.457463 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880ecc75bab446e39c687f3c8b740361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.457471 13246 ts_tablet_manager.cc:1909] T 6d8a9ecf47d7474c82758417449d940c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.461468 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43da1d12ce647dcbcd31263932b2f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.465456 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ef5a2799b447f992207452dbe39645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.469125 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 344) took 1994 ms. Trace:
I20240506 19:56:40.469679 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ace0ec43ba44060bcec2a50e1f48098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.469697 13246 rpcz_store.cc:276] 0506 19:56:38.474168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.474223 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:39.634846 (+1160623us) ts_tablet_manager.cc:1120] Deleting tablet 6d8a9ecf47d7474c82758417449d940c
0506 19:56:39.634856 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.393313 (+758457us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.425738 (+ 32425us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.469067 (+ 43329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160423}
I20240506 19:56:40.470023 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b022c086d141a19a9b35c4a8c8bef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.471812 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddae4b0d8d0943d99f54904266f293d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.472992 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260d7ab458794df496012cb80466f59c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:40.473563 13058 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.473681 13244 tablet_bootstrap.cc:492] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:40.473910 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b24fa4663244a88dac9ea99fe38f3a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:40.473863 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd3fc62e9614cf3bde0202ee6832bb3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:40.475286 13246 tablet_replica.cc:330] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.475839 13122 raft_consensus.cc:3037] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.483265 13246 raft_consensus.cc:2229] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.483304 13091 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 368) took 1086 ms. Trace:
I20240506 19:56:40.485349 12818 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 368) took 1043 ms. Trace:
I20240506 19:56:40.485952 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 287) took 2827 ms. Trace:
I20240506 19:56:40.487354 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f382a89f94e4fdf9b8266d049635329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.488106 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b022c086d141a19a9b35c4a8c8bef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.491681 13246 raft_consensus.cc:2258] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.493798 13122 raft_consensus.cc:2454] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:40.495186 13716 leader_election.cc:288] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.495105 13091 rpcz_store.cc:276] 0506 19:56:39.396325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.396372 (+ 47us) service_pool.cc:225] Handling call
0506 19:56:39.665210 (+268838us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.745524 (+ 80314us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.416172 (+670648us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.482377 (+ 66205us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.482511 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.495232 12895 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:40.495296 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba2365d00044c32a06f9017a207adf7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:40.496642 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373e04e93a5543f9934077006eaf2ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.496663 13629 leader_election.cc:288] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.501904 13246 ts_tablet_manager.cc:1875] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.504626 12824 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 366) took 1029 ms. Trace:
I20240506 19:56:40.504768 12824 rpcz_store.cc:276] 0506 19:56:39.403859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.649728 (+245869us) service_pool.cc:225] Handling call
0506 19:56:39.650228 (+ 500us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.650257 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.702836 (+ 52579us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.864386 (+161550us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.433357 (+568971us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 19:56:40.506630 13058 leader_election.cc:341] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.508183 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66f1e46cce47639fe644fb4ed2e5c0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:40.511412 13057 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.512063 13057 leader_election.cc:302] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:40.513621 13730 raft_consensus.cc:2726] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:40.513731 12921 leader_election.cc:302] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:40.523424 13629 raft_consensus.cc:2781] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 19:56:40.524219 13719 leader_election.cc:288] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.523034 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba2365d00044c32a06f9017a207adf7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:40.524755 12922 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.525417 13353 raft_consensus.cc:482] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.526616 13100 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 373) took 1116 ms. Trace:
W20240506 19:56:40.526841 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 64136.8us
I20240506 19:56:40.527875 12922 leader_election.cc:302] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:40.529436 13629 raft_consensus.cc:482] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:40.529250 13086 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 381) took 1065 ms. Trace:
I20240506 19:56:40.526912 13353 raft_consensus.cc:504] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:40.530488 12921 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.530575 13100 rpcz_store.cc:276] 0506 19:56:39.400035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.400086 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:39.748751 (+348665us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.748781 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.444961 (+696180us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.511146 (+ 66185us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.516058 (+ 4912us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.531167 13629 raft_consensus.cc:3037] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Advancing to term 2
I20240506 19:56:40.531701 12921 leader_election.cc:302] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:40.531677 13719 raft_consensus.cc:2781] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:40.532359 13096 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 377) took 1085 ms. Trace:
I20240506 19:56:40.532677 13086 rpcz_store.cc:276] 0506 19:56:39.451119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.451181 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:39.749238 (+298057us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.749266 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.444954 (+695688us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.516175 (+ 71221us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.516273 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.533483 13719 raft_consensus.cc:686] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:40.533478 12809 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 367) took 1015 ms. Trace:
I20240506 19:56:40.533625 12809 rpcz_store.cc:276] 0506 19:56:39.433320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.657622 (+224302us) service_pool.cc:225] Handling call
0506 19:56:39.658070 (+ 448us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.658096 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.661048 (+ 2952us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.878349 (+217301us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.448371 (+570022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.533773 13719 consensus_queue.cc:237] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.534830 13096 rpcz_store.cc:276] 0506 19:56:39.439806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.439877 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:39.753859 (+313982us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.753884 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.496654 (+742770us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.524671 (+ 28017us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.524812 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.535146 12827 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 365) took 1044 ms. Trace:
I20240506 19:56:40.535280 12827 rpcz_store.cc:276] 0506 19:56:39.403782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.645645 (+241863us) service_pool.cc:225] Handling call
0506 19:56:39.646117 (+ 472us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.646143 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.690702 (+ 44559us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.878628 (+187926us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.448445 (+569817us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.536440 13720 raft_consensus.cc:2726] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:40.539572 12922 leader_election.cc:341] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 92951843c8bd446aa86436b65d9eb793
I20240506 19:56:40.541009 13245 tablet_bootstrap.cc:492] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:40.538044 12921 leader_election.cc:341] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.541549 13484 ts_manager.cc:194] Registered new tserver with Master: 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.542676 13353 leader_election.cc:288] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:40.542186 13057 leader_election.cc:341] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.543248 13095 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 383) took 1082 ms. Trace:
I20240506 19:56:40.543149 13087 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 380) took 1085 ms. Trace:
I20240506 19:56:40.544270 13057 leader_election.cc:302] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:40.544700 13730 raft_consensus.cc:2726] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:40.545306 13095 rpcz_store.cc:276] 0506 19:56:39.456048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.851833 (+395785us) service_pool.cc:225] Handling call
0506 19:56:40.270616 (+418783us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:40.270650 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.489517 (+218867us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.538137 (+ 48620us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.538310 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.545735 12828 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 363) took 1046 ms. Trace:
I20240506 19:56:40.545877 12828 rpcz_store.cc:276] 0506 19:56:39.403596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.634801 (+231205us) service_pool.cc:225] Handling call
0506 19:56:39.635376 (+ 575us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.635415 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.759349 (+123934us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.882641 (+123292us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.450071 (+567430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.537380 12823 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 369) took 1010 ms. Trace:
I20240506 19:56:40.546357 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d819d1237f4798b4847fdf378cbc62" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:40.548734 13098 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 374) took 1116 ms. Trace:
I20240506 19:56:40.548979 12815 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 372) took 1004 ms. Trace:
I20240506 19:56:40.549119 12815 rpcz_store.cc:276] 0506 19:56:39.446122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.682869 (+236747us) service_pool.cc:225] Handling call
0506 19:56:39.683428 (+ 559us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.683459 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.745596 (+ 62137us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.885750 (+140154us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.450152 (+564402us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1494}
I20240506 19:56:40.545797 13087 rpcz_store.cc:276] 0506 19:56:39.448158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.448379 (+ 221us) service_pool.cc:225] Handling call
0506 19:56:39.749310 (+300931us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.753430 (+ 4120us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.450578 (+697148us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.532945 (+ 82367us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.533103 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.549664 13245 ts_tablet_manager.cc:1367] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.917s user 0.009s sys 0.009s
I20240506 19:56:40.549831 13098 rpcz_store.cc:276] 0506 19:56:39.432155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.432217 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:39.748858 (+316641us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.748893 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.520291 (+771398us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.548535 (+ 28244us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.548692 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.550781 13088 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 379) took 1099 ms. Trace:
I20240506 19:56:40.550961 13088 rpcz_store.cc:276] 0506 19:56:39.445921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.445981 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:39.753908 (+307927us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.754630 (+ 722us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.507286 (+752656us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.544984 (+ 37698us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.545157 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.553929 12810 rpcz_store.cc:276] 0506 19:56:39.403500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.620112 (+216612us) service_pool.cc:225] Handling call
0506 19:56:39.620695 (+ 583us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.620729 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.693466 (+ 72737us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.883044 (+189578us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.448520 (+565476us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 19:56:40.555178 12783 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 581aae0889654903933bef3bf1026954
W20240506 19:56:40.555395 12783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 74863us
I20240506 19:56:40.557346 12823 rpcz_store.cc:276] 0506 19:56:39.437484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.665420 (+227936us) service_pool.cc:225] Handling call
0506 19:56:39.665888 (+ 468us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.665925 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.669035 (+ 3110us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.878038 (+209003us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.448289 (+570251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.557729 13484 ts_manager.cc:194] Registered new tserver with Master: ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:40.562119 12818 rpcz_store.cc:276] 0506 19:56:39.435347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.661428 (+226081us) service_pool.cc:225] Handling call
0506 19:56:39.661868 (+ 440us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.661891 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:39.664965 (+ 3074us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:39.863443 (+198478us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.478564 (+615121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.563973 13242 rpcz_store.cc:276] 0506 19:56:37.658952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.659004 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:39.624374 (+1965370us) ts_tablet_manager.cc:1120] Deleting tablet d843fe82af1148f4a9045ce8b09ad11d
0506 19:56:39.624385 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:39.692787 (+ 68402us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.327450 (+634663us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.485911 (+158461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964546}
I20240506 19:56:40.564872 13244 tablet_bootstrap.cc:654] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.566376 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d843fe82af1148f4a9045ce8b09ad11d (table default.Table_bbc623ea167c48929e77cee2b3d07aaa [id=f6f5f08248a142d6929736e7872ae027]) successfully deleted
I20240506 19:56:40.569392 13085 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 382) took 1116 ms. Trace:
I20240506 19:56:40.569538 13085 rpcz_store.cc:276] 0506 19:56:39.452941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.453086 (+ 145us) service_pool.cc:225] Handling call
0506 19:56:39.749149 (+296063us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.749178 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.514635 (+765457us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.569109 (+ 54474us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.569319 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.570735 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d819d1237f4798b4847fdf378cbc62" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:40.572041 12781 leader_election.cc:341] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 581aae0889654903933bef3bf1026954
I20240506 19:56:40.572341 12781 leader_election.cc:302] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:40.573031 13716 raft_consensus.cc:2726] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:40.573264 13242 tablet_replica.cc:330] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:40.574905 13242 raft_consensus.cc:2229] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.575124 13242 raft_consensus.cc:2258] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.576476 13242 ts_tablet_manager.cc:1875] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.574139 13103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 372) took 1175 ms. Trace:
I20240506 19:56:40.579715 13103 rpcz_store.cc:276] 0506 19:56:39.399104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.401250 (+ 2146us) service_pool.cc:225] Handling call
0506 19:56:39.749072 (+347822us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.749126 (+ 54us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.562244 (+813118us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.573872 (+ 11628us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.574087 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.583271 13092 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 376) took 1146 ms. Trace:
I20240506 19:56:40.583391 13092 rpcz_store.cc:276] 0506 19:56:39.437157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.437209 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:39.753796 (+316587us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.753832 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.535495 (+781663us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.582974 (+ 47479us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.583207 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.566747 13245 raft_consensus.cc:348] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.584057 13245 raft_consensus.cc:374] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.584154 13245 raft_consensus.cc:729] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.585103 13245 consensus_queue.cc:260] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.585831 13245 ts_tablet_manager.cc:1398] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240506 19:56:40.592173 13246 ts_tablet_manager.cc:1888] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.592375 13246 log.cc:1196] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/27b24fa4663244a88dac9ea99fe38f3a
I20240506 19:56:40.592793 13246 ts_tablet_manager.cc:1909] T 27b24fa4663244a88dac9ea99fe38f3a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.577245 13104 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 369) took 1180 ms. Trace:
I20240506 19:56:40.593281 13104 rpcz_store.cc:276] 0506 19:56:39.396810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.396864 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:39.746006 (+349142us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.746050 (+ 44us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.564557 (+818507us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.576963 (+ 12406us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.577182 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.579536 13089 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 375) took 1144 ms. Trace:
I20240506 19:56:40.593852 13089 rpcz_store.cc:276] 0506 19:56:39.435091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.435152 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:39.754660 (+319508us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.754688 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.526462 (+771774us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.579248 (+ 52786us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.579471 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.594540 13245 tablet_bootstrap.cc:492] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.595547 13102 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56326 (request call id 378) took 1153 ms. Trace:
I20240506 19:56:40.595652 13102 rpcz_store.cc:276] 0506 19:56:39.442383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.442467 (+ 84us) service_pool.cc:225] Handling call
0506 19:56:39.748972 (+306505us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.749013 (+ 41us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.528399 (+779386us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.587781 (+ 59382us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.595493 (+ 7712us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.597404 13245 tablet_bootstrap.cc:654] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.599198 12820 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 374) took 1144 ms. Trace:
I20240506 19:56:40.599217 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 345) took 2120 ms. Trace:
I20240506 19:56:40.599509 13246 rpcz_store.cc:276] 0506 19:56:38.478728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.478780 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:40.470164 (+1991384us) ts_tablet_manager.cc:1120] Deleting tablet 27b24fa4663244a88dac9ea99fe38f3a
0506 19:56:40.470175 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.553941 (+ 83766us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.592005 (+ 38064us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.598905 (+ 6900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8576,"tablet-delete.queue_time_us":1991095}
I20240506 19:56:40.606297 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 346) took 1800 ms. Trace:
I20240506 19:56:40.606735 13246 rpcz_store.cc:276] 0506 19:56:38.805956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.806014 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:40.600089 (+1794075us) ts_tablet_manager.cc:1120] Deleting tablet 17da01620b8b41008421b977455e0c9a
0506 19:56:40.605925 (+ 5836us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.606148 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793773}
I20240506 19:56:40.607620 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 347) took 1800 ms. Trace:
I20240506 19:56:40.607772 13246 rpcz_store.cc:276] 0506 19:56:38.806980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.807037 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:40.607354 (+1800317us) ts_tablet_manager.cc:1120] Deleting tablet 2b5a46b91cb742a994496648fbf87aa4
0506 19:56:40.607366 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.607519 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800066}
I20240506 19:56:40.608755 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 348) took 1800 ms. Trace:
I20240506 19:56:40.608963 13246 rpcz_store.cc:276] 0506 19:56:38.807685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.807729 (+ 44us) service_pool.cc:225] Handling call
0506 19:56:40.608427 (+1800698us) ts_tablet_manager.cc:1120] Deleting tablet 971db6ac24d54efca95695ed1c90d4ed
0506 19:56:40.608462 (+ 35us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.608614 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800457}
I20240506 19:56:40.610113 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 349) took 1794 ms. Trace:
I20240506 19:56:40.610323 13246 rpcz_store.cc:276] 0506 19:56:38.815546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.815608 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:40.609732 (+1794124us) ts_tablet_manager.cc:1120] Deleting tablet 607b3fc2c17b4793af9b84c19066ec5e
0506 19:56:40.609745 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.609966 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793875}
I20240506 19:56:40.611380 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 358) took 1401 ms. Trace:
I20240506 19:56:40.611596 13246 rpcz_store.cc:276] 0506 19:56:39.209989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.210054 (+ 65us) service_pool.cc:225] Handling call
0506 19:56:40.611021 (+1400967us) ts_tablet_manager.cc:1120] Deleting tablet 1615f6167aae4dc096b9f563ab5e6215
0506 19:56:40.611033 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.611251 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1400645}
I20240506 19:56:40.612608 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 359) took 1394 ms. Trace:
W20240506 19:56:40.612695 12922 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.612818 13246 rpcz_store.cc:276] 0506 19:56:39.217725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.218007 (+ 282us) service_pool.cc:225] Handling call
0506 19:56:40.612293 (+1394286us) ts_tablet_manager.cc:1120] Deleting tablet d7be9788df8e47d285bd932d078c6a8b
0506 19:56:40.612307 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.612512 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394041}
I20240506 19:56:40.613862 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 360) took 1393 ms. Trace:
I20240506 19:56:40.614105 13246 rpcz_store.cc:276] 0506 19:56:39.220355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.220425 (+ 70us) service_pool.cc:225] Handling call
0506 19:56:40.613546 (+1393121us) ts_tablet_manager.cc:1120] Deleting tablet f8951bf4873e4382bcd42a52368fb376
0506 19:56:40.613557 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.613768 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1392883}
I20240506 19:56:40.615105 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 361) took 1392 ms. Trace:
I20240506 19:56:40.615336 13246 rpcz_store.cc:276] 0506 19:56:39.222762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.222816 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:40.614697 (+1391881us) ts_tablet_manager.cc:1120] Deleting tablet 85bc7cfd006f4d4ca95541dab6873d32
0506 19:56:40.614750 (+ 53us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.614965 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391038}
I20240506 19:56:40.616117 13246 tablet_replica.cc:330] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.616544 13246 raft_consensus.cc:2229] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:40.617225 13246 pending_rounds.cc:62] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:40.617399 13246 pending_rounds.cc:68] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744039705968640 op_type: NO_OP noop_request { }
I20240506 19:56:40.617707 13246 raft_consensus.cc:2866] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:40.618050 13246 raft_consensus.cc:2258] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.618106 13730 raft_consensus.cc:482] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.618261 13730 raft_consensus.cc:504] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.619096 13730 leader_election.cc:288] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.621474 13246 ts_tablet_manager.cc:1875] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.619741 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d819d1237f4798b4847fdf378cbc62" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:40.625257 13058 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.625779 12820 rpcz_store.cc:276] 0506 19:56:39.451202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.786257 (+335055us) service_pool.cc:225] Handling call
0506 19:56:40.448908 (+662651us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:40.448942 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.465199 (+ 16257us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.594502 (+129303us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.595447 (+ 945us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 19:56:40.627157 12921 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.627432 12921 leader_election.cc:302] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:40.629200 13719 raft_consensus.cc:2726] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:40.630313 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 61893us
W20240506 19:56:40.631516 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 54594.6us
I20240506 19:56:40.638828 13246 ts_tablet_manager.cc:1888] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:40.639202 13246 log.cc:1196] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/393073bca2a04d15b3e583700262f3e0
I20240506 19:56:40.640199 13246 ts_tablet_manager.cc:1909] T 393073bca2a04d15b3e583700262f3e0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.642309 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 362) took 1375 ms. Trace:
I20240506 19:56:40.642541 13246 rpcz_store.cc:276] 0506 19:56:39.266989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.267069 (+ 80us) service_pool.cc:225] Handling call
0506 19:56:40.615932 (+1348863us) ts_tablet_manager.cc:1120] Deleting tablet 393073bca2a04d15b3e583700262f3e0
0506 19:56:40.615973 (+ 41us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.631651 (+ 15678us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.638535 (+ 6884us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.642135 (+ 3600us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348509}
I20240506 19:56:40.643761 13246 tablet_replica.cc:330] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.644327 13246 raft_consensus.cc:2229] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.626597 12811 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 359) took 1223 ms. Trace:
I20240506 19:56:40.644757 13246 raft_consensus.cc:2258] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.637137 12814 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 361) took 1233 ms. Trace:
I20240506 19:56:40.644973 12814 rpcz_store.cc:276] 0506 19:56:39.403421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.511358 (+107937us) service_pool.cc:225] Handling call
0506 19:56:39.689223 (+177865us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.689249 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.413535 (+724286us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.563836 (+150301us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.637051 (+ 73215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.646170 13246 ts_tablet_manager.cc:1875] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.637411 12825 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 375) took 1183 ms. Trace:
I20240506 19:56:40.646584 12825 rpcz_store.cc:276] 0506 19:56:39.454169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.794951 (+340782us) service_pool.cc:225] Handling call
0506 19:56:40.450750 (+655799us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:40.450800 (+ 50us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.457156 (+ 6356us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.570388 (+113232us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.637378 (+ 66990us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.622126 12817 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 376) took 1162 ms. Trace:
I20240506 19:56:40.647485 12817 rpcz_store.cc:276] 0506 19:56:39.459412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.834091 (+374679us) service_pool.cc:225] Handling call
0506 19:56:40.449026 (+614935us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:40.450620 (+ 1594us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.469073 (+ 18453us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.621886 (+152813us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.622069 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.619273 12826 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 373) took 1169 ms. Trace:
I20240506 19:56:40.648913 12826 rpcz_store.cc:276] 0506 19:56:39.449323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.783009 (+333686us) service_pool.cc:225] Handling call
0506 19:56:40.448748 (+665739us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:40.448790 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.492404 (+ 43614us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.619076 (+126672us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.619232 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.644779 12811 rpcz_store.cc:276] 0506 19:56:39.403243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.448346 (+ 45103us) service_pool.cc:225] Handling call
0506 19:56:39.688575 (+240229us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.688605 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.349505 (+660900us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.626364 (+276859us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.626538 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.650405 13245 tablet_bootstrap.cc:492] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:40.650696 13245 ts_tablet_manager.cc:1367] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.032s
I20240506 19:56:40.654088 13245 raft_consensus.cc:348] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.654593 13245 raft_consensus.cc:374] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.654827 13245 raft_consensus.cc:729] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.655262 13245 consensus_queue.cc:260] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.655577 13242 ts_tablet_manager.cc:1888] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.655660 13246 ts_tablet_manager.cc:1888] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.655782 13246 log.cc:1196] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/0ba2365d00044c32a06f9017a207adf7
I20240506 19:56:40.656656 13246 ts_tablet_manager.cc:1909] T 0ba2365d00044c32a06f9017a207adf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.657179 13245 ts_tablet_manager.cc:1398] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240506 19:56:40.657474 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d819d1237f4798b4847fdf378cbc62" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:40.657749 12986 raft_consensus.cc:2454] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:40.658756 13057 leader_election.cc:302] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:40.659138 13730 raft_consensus.cc:2781] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:40.659235 13730 raft_consensus.cc:482] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:40.659302 13730 raft_consensus.cc:3037] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.662056 13730 raft_consensus.cc:504] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.662588 13245 tablet_bootstrap.cc:492] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.663417 13730 leader_election.cc:288] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.662086 12958 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41490 (request call id 387) took 1204 ms. Trace:
I20240506 19:56:40.670125 13246 tablet_replica.cc:330] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.670928 13246 raft_consensus.cc:2229] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:40.672178 13246 raft_consensus.cc:2258] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.672482 13245 tablet_bootstrap.cc:654] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.674575 13246 ts_tablet_manager.cc:1875] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.702028 13246 ts_tablet_manager.cc:1888] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:40.702781 13246 log.cc:1196] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/320e714ed6b64f2d84770e7182e4103a
I20240506 19:56:40.703258 13629 raft_consensus.cc:504] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.704442 13246 ts_tablet_manager.cc:1909] T 320e714ed6b64f2d84770e7182e4103a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.708168 13246 tablet_replica.cc:330] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.708958 13246 raft_consensus.cc:2229] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.709090 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d819d1237f4798b4847fdf378cbc62" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:40.709247 13629 leader_election.cc:288] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.709331 12986 raft_consensus.cc:3037] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.709738 13246 raft_consensus.cc:2258] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:40.711031 12922 leader_election.cc:341] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 92951843c8bd446aa86436b65d9eb793
I20240506 19:56:40.711834 13246 ts_tablet_manager.cc:1875] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.711977 12958 rpcz_store.cc:276] 0506 19:56:39.457366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.896346 (+438980us) service_pool.cc:225] Handling call
0506 19:56:40.270745 (+374399us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:40.270777 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.511248 (+240471us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.661843 (+150595us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.662007 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.715751 12986 raft_consensus.cc:2454] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:40.716269 13057 leader_election.cc:302] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:40.717720 13730 raft_consensus.cc:2781] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:40.719327 13730 raft_consensus.cc:686] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:40.719839 13730 consensus_queue.cc:237] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.730726 12515 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33592 (request call id 451) took 1211 ms. Trace:
I20240506 19:56:40.732203 12519 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33592 (request call id 452) took 1211 ms. Trace:
I20240506 19:56:40.660611 13716 raft_consensus.cc:482] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.733598 13246 ts_tablet_manager.cc:1888] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.733752 13246 log.cc:1196] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:40.733580 13716 raft_consensus.cc:504] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.733879 13242 log.cc:1196] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a223c33ba56c4ddbba46fedbce660ae2
I20240506 19:56:40.733913 13249 tablet_bootstrap.cc:492] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.734112 13249 ts_tablet_manager.cc:1367] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.470s user 0.026s sys 0.041s
I20240506 19:56:40.734154 13246 ts_tablet_manager.cc:1909] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.734488 13716 leader_election.cc:288] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.664191 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d819d1237f4798b4847fdf378cbc62" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:40.734848 13249 raft_consensus.cc:348] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.709975 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92951843c8bd446aa86436b65d9eb793" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:40.715549 13689 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.390s user 0.014s sys 0.048s
I20240506 19:56:40.735190 12847 raft_consensus.cc:3037] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Advancing to term 2
I20240506 19:56:40.735416 13242 ts_tablet_manager.cc:1909] T a223c33ba56c4ddbba46fedbce660ae2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:40.738010 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 288) took 3077 ms. Trace:
I20240506 19:56:40.739900 13249 raft_consensus.cc:374] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.740020 13249 raft_consensus.cc:729] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.740406 13249 consensus_queue.cc:260] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.741426 13249 ts_tablet_manager.cc:1398] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240506 19:56:40.741770 13249 tablet_bootstrap.cc:492] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:40.744884 13058 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.746505 13249 tablet_bootstrap.cc:654] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.738512 12847 raft_consensus.cc:2454] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 2.
W20240506 19:56:40.744110 12783 leader_election.cc:341] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a16704567ef6454eaef886fb9cd599ce
W20240506 19:56:40.749986 12781 leader_election.cc:341] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:40.750154 12781 leader_election.cc:302] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:40.750782 12921 leader_election.cc:302] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:40.751107 13629 raft_consensus.cc:2781] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 19:56:40.751289 13629 raft_consensus.cc:686] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 2 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:40.751590 13629 consensus_queue.cc:237] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.753108 12819 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50504 (request call id 360) took 1349 ms. Trace:
I20240506 19:56:40.753209 12819 rpcz_store.cc:276] 0506 19:56:39.403324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.500716 (+ 97392us) service_pool.cc:225] Handling call
0506 19:56:39.709378 (+208662us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0506 19:56:39.709416 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0506 19:56:40.668979 (+959563us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.752883 (+ 83904us) tablet_replica.cc:205] Creating consensus instance
0506 19:56:40.753051 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:40.754492 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 6156c6f582b747de973b357b6d071a46
I20240506 19:56:40.754591 13243 ts_tablet_manager.cc:1888] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:40.754873 13716 raft_consensus.cc:2726] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:40.755267 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 6156c6f582b747de973b357b6d071a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.754751 13243 log.cc:1196] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a80834187f874661b611b43065eee943
I20240506 19:56:40.756021 13243 ts_tablet_manager.cc:1909] T a80834187f874661b611b43065eee943 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:40.755443 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6156c6f582b747de973b357b6d071a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:40.758744 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd57563cc6c140c184d1725a2da6a475 consensus state has no leader
I20240506 19:56:40.750058 13242 rpcz_store.cc:276] 0506 19:56:37.660082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.660131 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:40.564352 (+2904221us) ts_tablet_manager.cc:1120] Deleting tablet a223c33ba56c4ddbba46fedbce660ae2
0506 19:56:40.571689 (+ 7337us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.580801 (+ 9112us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.655436 (+ 74635us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.737944 (+ 82508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2904028}
I20240506 19:56:40.759315 13242 tablet_replica.cc:330] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:40.759582 13242 raft_consensus.cc:2229] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.759815 13242 raft_consensus.cc:2258] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.760592 13242 ts_tablet_manager.cc:1875] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.756837 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 256) took 3852 ms. Trace:
I20240506 19:56:40.761445 13243 rpcz_store.cc:276] 0506 19:56:36.904558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.904619 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:40.356779 (+3452160us) ts_tablet_manager.cc:1120] Deleting tablet a80834187f874661b611b43065eee943
0506 19:56:40.356794 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.725549 (+368755us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.752742 (+ 27193us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.756776 (+ 4034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3451897}
I20240506 19:56:40.761881 13243 tablet_replica.cc:330] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:40.762115 13243 raft_consensus.cc:2229] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.762308 13243 raft_consensus.cc:2258] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.762923 13243 ts_tablet_manager.cc:1875] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.763981 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a80834187f874661b611b43065eee943 (table default.Table_2b51a8ac571347a896b8741066c330c4 [id=c18e335476d443f9949f5f38fa4550ae]) successfully deleted
I20240506 19:56:40.730865 12515 rpcz_store.cc:276] 0506 19:56:39.518369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.721138 (+202769us) service_pool.cc:225] Handling call
0506 19:56:39.725092 (+ 3954us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x616000021d54. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 00007f8feed4915d 00007f8feed49253 00007f8feba7ad41 00007f8ff366734d 00007f8ff37f3cb3 00007f8fea7e52a6 00007f8fea7e4fd5 00007f8fe6f1104e 00007f8fe6f0ff2e 00007f8fe6f16405 00007f8fe6f16d2f
0506 19:56:40.277722 (+552630us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 19:56:40.278191 (+ 469us) catalog_manager.cc:2118] Created new table and tablet info
0506 19:56:40.421592 (+143401us) catalog_manager.cc:2178] Wrote table and tablets to system table
0506 19:56:40.729532 (+307940us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0506 19:56:40.729695 (+ 163us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:40.279466 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:40.279562 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:40.279566 (+ 4us) write_op.cc:435] Acquired schema lock
0506 19:56:40.279570 (+ 4us) tablet.cc:660] Decoding operations
0506 19:56:40.279646 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:40.279653 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:40.279658 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:40.279734 (+ 76us) tablet.cc:699] Row locks acquired
0506 19:56:40.279736 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:40.279762 (+ 26us) write_op.cc:270] Start()
0506 19:56:40.279783 (+ 21us) write_op.cc:276] Timestamp: P: 1715025400279693 usec, L: 0
0506 19:56:40.279786 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:40.279908 (+ 122us) log.cc:842] Serialized 433 byte log entry
0506 19:56:40.419693 (+139785us) op_driver.cc:444] REPLICATION: finished
0506 19:56:40.420789 (+ 1096us) write_op.cc:301] APPLY: starting
0506 19:56:40.420800 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:40.420833 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:40.420856 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:40.420942 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:40.420975 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 19:56:40.420984 (+ 9us) write_op.cc:312] APPLY: finished
0506 19:56:40.421056 (+ 72us) log.cc:842] Serialized 29 byte log entry
0506 19:56:40.421099 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.421114 (+ 15us) write_op.cc:454] Released schema lock
0506 19:56:40.421120 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:40.421165 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.421169 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":8500992,"child_traces":[["op",{"apply.queue_time_us":1067,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":419,"num_ops":2,"prepare.queue_time_us":890,"prepare.run_cpu_time_us":484,"prepare.run_wall_time_us":483,"replication_time_us":139893}]]}
I20240506 19:56:40.766840 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a223c33ba56c4ddbba46fedbce660ae2 (table default.Table_218a52f567cf4031bee1b7834387ac84 [id=779ea3288b6d40ceb9636fdde3697472]) successfully deleted
I20240506 19:56:40.772078 13245 tablet_bootstrap.cc:492] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:40.772281 13245 ts_tablet_manager.cc:1367] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.110s user 0.001s sys 0.036s
I20240506 19:56:40.773059 13245 raft_consensus.cc:348] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.773283 13245 raft_consensus.cc:374] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.773384 13245 raft_consensus.cc:729] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.773579 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e737c1fa78974f05a022649a900e2636" } modify_external_catalogs: true
I20240506 19:56:40.773608 13245 consensus_queue.cc:260] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.773777 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e737c1fa78974f05a022649a900e2636" } modify_external_catalogs: true
I20240506 19:56:40.774103 13245 ts_tablet_manager.cc:1398] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:40.773398 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 6d8a9ecf47d7474c82758417449d940c (table default.Table_8de44207890c4ceb8542f805a6f9b8da [id=70fc9e614eab4b11baeefbbf7127f540]) successfully deleted
I20240506 19:56:40.774531 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc" } modify_external_catalogs: true
I20240506 19:56:40.776557 13242 ts_tablet_manager.cc:1888] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.776695 13242 log.cc:1196] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/8596757ef75e433fb801f7679db5fefb
I20240506 19:56:40.777091 13245 tablet_bootstrap.cc:492] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.780551 13245 tablet_bootstrap.cc:654] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.780548 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc" } modify_external_catalogs: true
I20240506 19:56:40.781558 13242 ts_tablet_manager.cc:1909] T 8596757ef75e433fb801f7679db5fefb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:40.781560 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_80e93d6f7329468c85d894226c0390c4" } new_table_name: "default.Table_b33a15919aac48e699db676197ad1e75" modify_external_catalogs: true
I20240506 19:56:40.732321 12519 rpcz_store.cc:276] 0506 19:56:39.518733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.721419 (+202686us) service_pool.cc:225] Handling call
0506 19:56:40.277215 (+555796us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 19:56:40.277760 (+ 545us) catalog_manager.cc:2118] Created new table and tablet info
0506 19:56:40.420548 (+142788us) catalog_manager.cc:2178] Wrote table and tablets to system table
0506 19:56:40.729457 (+308909us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0506 19:56:40.730639 (+ 1182us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:40.278920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:40.279026 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:40.279031 (+ 5us) write_op.cc:435] Acquired schema lock
0506 19:56:40.279034 (+ 3us) tablet.cc:660] Decoding operations
0506 19:56:40.279104 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:40.279112 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:40.279116 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:40.279186 (+ 70us) tablet.cc:699] Row locks acquired
0506 19:56:40.279188 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:40.279217 (+ 29us) write_op.cc:270] Start()
0506 19:56:40.279247 (+ 30us) write_op.cc:276] Timestamp: P: 1715025400279147 usec, L: 0
0506 19:56:40.279251 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:40.279379 (+ 128us) log.cc:842] Serialized 433 byte log entry
0506 19:56:40.419606 (+140227us) op_driver.cc:444] REPLICATION: finished
0506 19:56:40.419913 (+ 307us) write_op.cc:301] APPLY: starting
0506 19:56:40.419933 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:40.419985 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:40.420020 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:40.420165 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:40.420205 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 19:56:40.420213 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:40.420351 (+ 138us) log.cc:842] Serialized 29 byte log entry
0506 19:56:40.420423 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.420449 (+ 26us) write_op.cc:454] Released schema lock
0506 19:56:40.420457 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:40.420699 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.420705 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":247,"apply.run_cpu_time_us":673,"apply.run_wall_time_us":847,"num_ops":2,"prepare.queue_time_us":638,"prepare.run_cpu_time_us":514,"prepare.run_wall_time_us":514,"replication_time_us":140327}]]}
I20240506 19:56:40.784912 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d819d1237f4798b4847fdf378cbc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.785177 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 289) took 3124 ms. Trace:
I20240506 19:56:40.785326 13242 rpcz_store.cc:276] 0506 19:56:37.660507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.662949 (+ 2442us) service_pool.cc:225] Handling call
0506 19:56:40.759233 (+3096284us) ts_tablet_manager.cc:1120] Deleting tablet 8596757ef75e433fb801f7679db5fefb
0506 19:56:40.759243 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.766101 (+ 6858us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.776415 (+ 10314us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.785135 (+ 8720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3096058}
I20240506 19:56:40.785392 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4d65e20e3d924618a805ec2be5bb9b29" } modify_external_catalogs: true
I20240506 19:56:40.785564 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4d65e20e3d924618a805ec2be5bb9b29" } modify_external_catalogs: true
I20240506 19:56:40.785876 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 8596757ef75e433fb801f7679db5fefb (table default.Table_73b96b599455444fa8fead40db334a12 [id=a76c2b5f25a24c55a8379608a9857eda]) successfully deleted
I20240506 19:56:40.785950 13242 tablet_replica.cc:330] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:40.786281 13242 raft_consensus.cc:2229] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.786597 13242 raft_consensus.cc:2258] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.787627 13242 ts_tablet_manager.cc:1875] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.787638 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ff9dd351a2594431addcdc137481cd02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:40.788332 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff9dd351a2594431addcdc137481cd02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.790712 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 27b24fa4663244a88dac9ea99fe38f3a (table default.Table_1f7a4e714f4a4412a5843504c95b87d7 [id=9844f952118d4225a5a7489527a6a703]) successfully deleted
W20240506 19:56:40.791232 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 17da01620b8b41008421b977455e0c9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17da01620b8b41008421b977455e0c9a already in progress: creating tablet
W20240506 19:56:40.791572 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2b5a46b91cb742a994496648fbf87aa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b5a46b91cb742a994496648fbf87aa4 already in progress: creating tablet
W20240506 19:56:40.791970 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 971db6ac24d54efca95695ed1c90d4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 971db6ac24d54efca95695ed1c90d4ed already in progress: creating tablet
W20240506 19:56:40.792255 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 607b3fc2c17b4793af9b84c19066ec5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607b3fc2c17b4793af9b84c19066ec5e already in progress: creating tablet
W20240506 19:56:40.792572 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1615f6167aae4dc096b9f563ab5e6215 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1615f6167aae4dc096b9f563ab5e6215 already in progress: creating tablet
W20240506 19:56:40.797396 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d7be9788df8e47d285bd932d078c6a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7be9788df8e47d285bd932d078c6a8b already in progress: creating tablet
I20240506 19:56:40.797396 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bd4b7877d06746518f96f0601873f10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:40.797778 13245 tablet_bootstrap.cc:492] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:40.797947 13245 ts_tablet_manager.cc:1367] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
W20240506 19:56:40.798154 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f8951bf4873e4382bcd42a52368fb376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8951bf4873e4382bcd42a52368fb376 already in progress: creating tablet
W20240506 19:56:40.798447 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 85bc7cfd006f4d4ca95541dab6873d32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85bc7cfd006f4d4ca95541dab6873d32 already in progress: creating tablet
I20240506 19:56:40.798741 13245 raft_consensus.cc:348] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.798965 13245 raft_consensus.cc:374] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:40.799038 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd4b7877d06746518f96f0601873f10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.799134 13245 raft_consensus.cc:729] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.799424 13245 consensus_queue.cc:260] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.799563 13249 tablet_bootstrap.cc:492] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.799703 13249 ts_tablet_manager.cc:1367] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.011s
I20240506 19:56:40.800432 13249 raft_consensus.cc:348] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.800645 13249 raft_consensus.cc:374] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.800752 13249 raft_consensus.cc:729] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.801021 13249 consensus_queue.cc:260] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.801262 13245 ts_tablet_manager.cc:1398] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:40.801569 13249 ts_tablet_manager.cc:1398] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:40.801594 13245 tablet_bootstrap.cc:492] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.801857 13249 tablet_bootstrap.cc:492] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:40.803416 13249 tablet_bootstrap.cc:654] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.805171 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 393073bca2a04d15b3e583700262f3e0 (table default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c [id=fd3051399ebf4aa2b31bbd709b120d48]) successfully deleted
I20240506 19:56:40.805604 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 0ba2365d00044c32a06f9017a207adf7 (table default.Table_24bc90d17d404ca3884c13c02624e651 [id=7a329c37e7684bac8bf008b1c5e53e1c]) successfully deleted
I20240506 19:56:40.806802 13244 tablet_bootstrap.cc:492] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:40.810496 13245 tablet_bootstrap.cc:654] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:40.811614 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4095f9fc28ff48f68064ec8318ff7b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4095f9fc28ff48f68064ec8318ff7b51 already in progress: creating tablet
W20240506 19:56:40.811939 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 997ecba7fbae41178a25bf7af4d152e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 997ecba7fbae41178a25bf7af4d152e1 already in progress: creating tablet
W20240506 19:56:40.812179 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 already in progress: creating tablet
I20240506 19:56:40.812366 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 320e714ed6b64f2d84770e7182e4103a (table default.Table_8be586b55bf2436698bfeb2f94fb465d [id=090dfa9462424be1b846e1cb05569156]) successfully deleted
I20240506 19:56:40.812439 12520 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33592 (request call id 459) took 1245 ms. Trace:
I20240506 19:56:40.812569 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 8f66f1e46cce47639fe644fb4ed2e5c0 (table default.Table_ca61f3a9aefa4fba8fea32d717de3fb1 [id=bf75c039ad324e50a2ccb889439e2b4c]) successfully deleted
W20240506 19:56:40.812749 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2519c03738864f6ab5dcfab7bc3fc8aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2519c03738864f6ab5dcfab7bc3fc8aa already in progress: creating tablet
I20240506 19:56:40.813001 13242 ts_tablet_manager.cc:1888] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.812546 12520 rpcz_store.cc:276] 0506 19:56:39.567324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.881645 (+314321us) service_pool.cc:225] Handling call
0506 19:56:40.294832 (+413187us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 19:56:40.295385 (+ 553us) catalog_manager.cc:2118] Created new table and tablet info
0506 19:56:40.812141 (+516756us) catalog_manager.cc:2178] Wrote table and tablets to system table
0506 19:56:40.812282 (+ 141us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0506 19:56:40.812382 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:40.318209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:40.318334 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:40.318340 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:40.318342 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:40.318455 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:40.318470 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:40.318477 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:40.318565 (+ 88us) tablet.cc:699] Row locks acquired
0506 19:56:40.318568 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:40.318608 (+ 40us) write_op.cc:270] Start()
0506 19:56:40.318643 (+ 35us) write_op.cc:276] Timestamp: P: 1715025400318528 usec, L: 0
0506 19:56:40.318647 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:40.318872 (+ 225us) log.cc:842] Serialized 433 byte log entry
0506 19:56:40.811317 (+492445us) op_driver.cc:444] REPLICATION: finished
0506 19:56:40.811434 (+ 117us) write_op.cc:301] APPLY: starting
0506 19:56:40.811445 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:40.811484 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:40.811540 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:40.811721 (+ 181us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:40.811754 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 19:56:40.811762 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:40.811856 (+ 94us) log.cc:842] Serialized 29 byte log entry
0506 19:56:40.811938 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.811956 (+ 18us) write_op.cc:454] Released schema lock
0506 19:56:40.811963 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:40.812018 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.812021 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"apply.run_cpu_time_us":630,"apply.run_wall_time_us":630,"num_ops":2,"prepare.queue_time_us":22233,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":752,"replication_time_us":492649}]]}
I20240506 19:56:40.813139 13242 log.cc:1196] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/30cb3cb7cbe14cc9bf1bfd07227f538e
I20240506 19:56:40.813376 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 6156c6f582b747de973b357b6d071a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.813647 13246 tablet_replica.cc:330] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.813868 13246 raft_consensus.cc:2229] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.814076 13246 raft_consensus.cc:2258] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.814641 13246 ts_tablet_manager.cc:1875] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.815763 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 034d6309de0740d0a5806b3847e334cb from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:40.817031 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 838d9e7986164ffd916d2d842dc61f6e consensus state has no leader
I20240506 19:56:40.817312 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 5dc0ac37f67942fc8a31d8d43c839246 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:40.819655 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:40.820128 13249 tablet_bootstrap.cc:492] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.820293 13249 ts_tablet_manager.cc:1367] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240506 19:56:40.820407 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_632a267fe67b4ac1af1a9882527ea33c" } new_table_name: "default.Table_ed6b424598be490999d5da821583c6b6" modify_external_catalogs: true
I20240506 19:56:40.820727 13719 raft_consensus.cc:482] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.820888 13719 raft_consensus.cc:504] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.820964 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5344cd78a3dd4575a939a926d7e1eb16" } modify_external_catalogs: true
I20240506 19:56:40.821121 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5344cd78a3dd4575a939a926d7e1eb16" } modify_external_catalogs: true
I20240506 19:56:40.821563 13719 leader_election.cc:288] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.822685 13246 ts_tablet_manager.cc:1888] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240506 19:56:40.822852 13246 log.cc:1196] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/6156c6f582b747de973b357b6d071a46
I20240506 19:56:40.822733 13249 raft_consensus.cc:348] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.822947 13249 raft_consensus.cc:374] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.823031 13249 raft_consensus.cc:729] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.823370 13246 ts_tablet_manager.cc:1909] T 6156c6f582b747de973b357b6d071a46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.823540 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ece0ad4df247eb9d380ed1c755ac99" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:40.823602 13249 consensus_queue.cc:260] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.810201 13244 ts_tablet_manager.cc:1367] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.337s user 0.013s sys 0.045s
I20240506 19:56:40.823810 13122 raft_consensus.cc:2454] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:40.824265 12922 leader_election.cc:302] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:40.823390 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ece0ad4df247eb9d380ed1c755ac99" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:40.824545 13719 raft_consensus.cc:2781] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:40.824633 13719 raft_consensus.cc:482] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:40.824743 13719 raft_consensus.cc:3037] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.813508 13242 ts_tablet_manager.cc:1909] T 30cb3cb7cbe14cc9bf1bfd07227f538e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:40.825759 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 290) took 3162 ms. Trace:
I20240506 19:56:40.825920 13242 rpcz_store.cc:276] 0506 19:56:37.663383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.663429 (+ 46us) service_pool.cc:225] Handling call
0506 19:56:40.785877 (+3122448us) ts_tablet_manager.cc:1120] Deleting tablet 30cb3cb7cbe14cc9bf1bfd07227f538e
0506 19:56:40.785889 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.794178 (+ 8289us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.812871 (+ 18693us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.825699 (+ 12828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3122230}
I20240506 19:56:40.826622 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 30cb3cb7cbe14cc9bf1bfd07227f538e (table default.Table_b8d6b8a6bab14e7bab5407af3dfd04bc [id=8b4499b016334533ab4366c2a4b1bd72]) successfully deleted
I20240506 19:56:40.827132 13242 tablet_replica.cc:330] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:40.827363 13242 raft_consensus.cc:2229] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.827606 13719 raft_consensus.cc:504] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.827960 13242 raft_consensus.cc:2258] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.828673 13242 ts_tablet_manager.cc:1875] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:40.829535 12921 leader_election.cc:341] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.829221 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ece0ad4df247eb9d380ed1c755ac99" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:40.818260 13491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
W20240506 19:56:40.830191 12921 leader_election.cc:341] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.830267 13491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:40.830651 13719 leader_election.cc:288] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.831132 13249 ts_tablet_manager.cc:1398] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:40.831176 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ece0ad4df247eb9d380ed1c755ac99" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:40.831404 13122 raft_consensus.cc:3037] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.831514 13249 tablet_bootstrap.cc:492] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:40.833477 13249 tablet_bootstrap.cc:654] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.833782 13244 raft_consensus.cc:348] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.834013 13244 raft_consensus.cc:374] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.834087 13244 raft_consensus.cc:729] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.834477 13122 raft_consensus.cc:2454] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:40.834956 12922 leader_election.cc:302] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:40.835237 13719 raft_consensus.cc:2781] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:40.835392 13245 tablet_bootstrap.cc:492] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:40.835395 13719 raft_consensus.cc:686] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:40.835546 13245 ts_tablet_manager.cc:1367] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.001s
I20240506 19:56:40.835656 13719 consensus_queue.cc:237] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.836256 13245 raft_consensus.cc:348] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.836467 13245 raft_consensus.cc:374] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.836601 13245 raft_consensus.cc:729] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.836896 13245 consensus_queue.cc:260] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.837715 13245 ts_tablet_manager.cc:1398] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240506 19:56:40.838619 13244 consensus_queue.cc:260] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.843581 13244 ts_tablet_manager.cc:1398] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240506 19:56:40.844626 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d819d1237f4798b4847fdf378cbc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.844940 13246 tablet_replica.cc:330] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.845192 13246 raft_consensus.cc:2229] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:40.845512 13246 pending_rounds.cc:62] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:40.845608 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.845616 13246 pending_rounds.cc:68] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744041352777728 op_type: NO_OP noop_request { }
I20240506 19:56:40.845944 13246 raft_consensus.cc:2866] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:40.846086 13246 raft_consensus.cc:2258] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.846382 13244 tablet_bootstrap.cc:492] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:40.846350 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.847507 13246 ts_tablet_manager.cc:1875] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.848209 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_44e570af6caa4a749207a8c828639acf [id=ee08d3829f304e16b5b8994259bb11b0]) successfully deleted
W20240506 19:56:40.849927 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f382a89f94e4fdf9b8266d049635329 consensus state has no leader
I20240506 19:56:40.852721 12819 tablet_service.cc:1467] Processing CreateTablet for tablet ff3049a9fa8049e5b84230e5e6e8245c (DEFAULT_TABLE table=default.Table_c0ebfdd5c1cb4085babb98af4665cf5d [id=1904607de54e494e9474ddf093d019f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.853211 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3049a9fa8049e5b84230e5e6e8245c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.854004 12952 tablet_service.cc:1467] Processing CreateTablet for tablet ff3049a9fa8049e5b84230e5e6e8245c (DEFAULT_TABLE table=default.Table_c0ebfdd5c1cb4085babb98af4665cf5d [id=1904607de54e494e9474ddf093d019f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.854455 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3049a9fa8049e5b84230e5e6e8245c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:40.856288 12515 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
I20240506 19:56:40.860551 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 1a6e0126d9854215a0e174a76c0a0391 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:40.860925 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 581aae0889654903933bef3bf1026954 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f56d2180b1472a8168fb3a63589065) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.861261 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e3f798d6484d4fabd55f9b4c98a911 with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff5fed57d64244894f4af1f6067e85) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.864207 13244 tablet_bootstrap.cc:654] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.864634 13090 tablet_service.cc:1467] Processing CreateTablet for tablet ff3049a9fa8049e5b84230e5e6e8245c (DEFAULT_TABLE table=default.Table_c0ebfdd5c1cb4085babb98af4665cf5d [id=1904607de54e494e9474ddf093d019f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:40.865667 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3049a9fa8049e5b84230e5e6e8245c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:40.874852 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46916e778bf34188959e1cfff084bffc consensus state has no leader
I20240506 19:56:40.875691 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5dc0ac37f67942fc8a31d8d43c839246
I20240506 19:56:40.876188 12518 catalog_manager.cc:6554] ReplaceTablet: tablet 5dc0ac37f67942fc8a31d8d43c839246 deleted and replaced by tablet c8ece0bd3a3b47ef88a0b4910f0efae9
I20240506 19:56:40.877014 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc0ac37f67942fc8a31d8d43c839246 with delete_type TABLET_DATA_DELETED (replaced by tablet c8ece0bd3a3b47ef88a0b4910f0efae9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.880299 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b9c18afe9e154517909885ea89fefcc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:40.880911 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9c18afe9e154517909885ea89fefcc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.881670 12519 catalog_manager.cc:5484] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:40.844779 13245 tablet_bootstrap.cc:492] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.877404 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc0ac37f67942fc8a31d8d43c839246 with delete_type TABLET_DATA_DELETED (replaced by tablet c8ece0bd3a3b47ef88a0b4910f0efae9) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.884248 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet b38453bf678748399339806d12993c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.884267 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc0ac37f67942fc8a31d8d43c839246 with delete_type TABLET_DATA_DELETED (replaced by tablet c8ece0bd3a3b47ef88a0b4910f0efae9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.885728 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:40.887527 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_15b63528dc5649619d25b71595773037"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:40.890877 13245 tablet_bootstrap.cc:654] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:40.893412 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15b63528dc5649619d25b71595773037 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.899029 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f2954b0b8209472b821fba70c582d658" } new_table_name: "default.Table_2f9d2a9b611448949d0a7fc4f5146078" modify_external_catalogs: true
I20240506 19:56:40.909634 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:56:40.910938 13246 ts_tablet_manager.cc:1888] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:40.911096 13246 log.cc:1196] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/30d819d1237f4798b4847fdf378cbc62
I20240506 19:56:40.911445 13249 tablet_bootstrap.cc:492] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.911484 13246 ts_tablet_manager.cc:1909] T 30d819d1237f4798b4847fdf378cbc62 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:40.911609 13249 ts_tablet_manager.cc:1367] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.021s
I20240506 19:56:40.912391 13249 raft_consensus.cc:348] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.912611 13249 raft_consensus.cc:374] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.912701 13249 raft_consensus.cc:729] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.912632 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dd9774c013d441f4b9fc4a0afdf7daa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:40.912940 13249 consensus_queue.cc:260] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:40.913215 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd9774c013d441f4b9fc4a0afdf7daa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:40.913538 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 034d6309de0740d0a5806b3847e334cb
I20240506 19:56:40.913648 13249 ts_tablet_manager.cc:1398] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:40.913904 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 034d6309de0740d0a5806b3847e334cb deleted and replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b
I20240506 19:56:40.913939 13249 tablet_bootstrap.cc:492] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:40.914142 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet b529bb9362924caca399491e7d9e03cd from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:40.915886 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 034d6309de0740d0a5806b3847e334cb with delete_type TABLET_DATA_DELETED (replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.916204 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.918130 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 034d6309de0740d0a5806b3847e334cb with delete_type TABLET_DATA_DELETED (replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.919953 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.920428 13249 tablet_bootstrap.cc:654] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.920601 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.920989 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 034d6309de0740d0a5806b3847e334cb with delete_type TABLET_DATA_DELETED (replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.921370 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 30d819d1237f4798b4847fdf378cbc62 (table default.Table_9d121b9e240b43cfbfea349bf45ef3a7 [id=332ee4f1670e4ddba0deb90d49e7b964]) successfully deleted
I20240506 19:56:40.921502 13242 ts_tablet_manager.cc:1888] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.921725 13242 log.cc:1196] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/64b7e42061cd487b8517447eb59ea8a1
I20240506 19:56:40.923007 13243 ts_tablet_manager.cc:1888] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:40.923247 13243 log.cc:1196] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/09856d2dd66043fbab08966d04e7b992
I20240506 19:56:40.926992 13243 ts_tablet_manager.cc:1909] T 09856d2dd66043fbab08966d04e7b992 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:40.927841 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 255) took 4023 ms. Trace:
I20240506 19:56:40.928020 13243 rpcz_store.cc:276] 0506 19:56:36.904402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:36.904989 (+ 587us) service_pool.cc:225] Handling call
0506 19:56:40.761755 (+3856766us) ts_tablet_manager.cc:1120] Deleting tablet 09856d2dd66043fbab08966d04e7b992
0506 19:56:40.761768 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.878072 (+116304us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.922828 (+ 44756us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.927804 (+ 4976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3856508}
I20240506 19:56:40.928712 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 257) took 3875 ms. Trace:
I20240506 19:56:40.922392 13730 raft_consensus.cc:482] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.929877 13730 raft_consensus.cc:504] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.921988 13246 tablet_replica.cc:330] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:40.930586 13242 ts_tablet_manager.cc:1909] T 64b7e42061cd487b8517447eb59ea8a1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:40.930740 13730 leader_election.cc:288] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:40.931293 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a3bed0eb234e21b890242e9d23a5b4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:40.931735 13244 tablet_bootstrap.cc:492] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:56:40.931790 13058 leader_election.cc:341] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:40.932109 13244 ts_tablet_manager.cc:1367] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.005s
I20240506 19:56:40.932495 13246 raft_consensus.cc:2229] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.932755 13246 raft_consensus.cc:2258] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.933970 13245 tablet_bootstrap.cc:492] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:40.934124 13245 ts_tablet_manager.cc:1367] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20240506 19:56:40.934945 13245 raft_consensus.cc:348] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.935168 13245 raft_consensus.cc:374] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.935263 13245 raft_consensus.cc:729] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.935480 13244 raft_consensus.cc:348] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.935654 13245 consensus_queue.cc:260] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.936192 13245 ts_tablet_manager.cc:1398] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:40.936619 13245 tablet_bootstrap.cc:492] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.938498 13245 tablet_bootstrap.cc:654] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.935801 13244 raft_consensus.cc:374] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.939177 13244 raft_consensus.cc:729] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.939498 13244 consensus_queue.cc:260] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.940358 13244 ts_tablet_manager.cc:1398] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240506 19:56:40.942740 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 291) took 3278 ms. Trace:
I20240506 19:56:40.943130 13244 tablet_bootstrap.cc:492] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:40.945466 13244 tablet_bootstrap.cc:654] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.946556 13242 rpcz_store.cc:276] 0506 19:56:37.663827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.663868 (+ 41us) service_pool.cc:225] Handling call
0506 19:56:40.827051 (+3163183us) ts_tablet_manager.cc:1120] Deleting tablet 64b7e42061cd487b8517447eb59ea8a1
0506 19:56:40.827061 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.889786 (+ 62725us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.921359 (+ 31573us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:40.942696 (+ 21337us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5973,"tablet-delete.queue_time_us":3162946}
I20240506 19:56:40.950723 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 64b7e42061cd487b8517447eb59ea8a1 (table default.Table_a84eed58b8c14359ad388d769fac448a [id=8bd1845a4f4a437781f9b4535852aa5c]) successfully deleted
I20240506 19:56:40.950871 13242 tablet_replica.cc:330] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:40.951102 13242 raft_consensus.cc:2229] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.951285 13242 raft_consensus.cc:2258] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.951948 13242 ts_tablet_manager.cc:1875] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.953694 13245 tablet_bootstrap.cc:492] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:40.953868 13245 ts_tablet_manager.cc:1367] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240506 19:56:40.954401 13246 ts_tablet_manager.cc:1875] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:40.954558 13245 raft_consensus.cc:348] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.955143 13245 raft_consensus.cc:374] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.955221 13245 raft_consensus.cc:729] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.955585 13245 consensus_queue.cc:260] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.956090 13245 ts_tablet_manager.cc:1398] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:40.956399 13245 tablet_bootstrap.cc:492] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:40.962038 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bece304e0ba048ecbb2d92aa71fdf707
I20240506 19:56:40.962580 12514 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 469) took 1032 ms. Trace:
I20240506 19:56:40.962833 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.931421 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a3bed0eb234e21b890242e9d23a5b4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:40.934386 13249 tablet_bootstrap.cc:492] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.962707 12514 rpcz_store.cc:276] 0506 19:56:39.930185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.784241 (+854056us) service_pool.cc:225] Handling call
0506 19:56:40.785678 (+ 1437us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:40.785851 (+ 173us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:40.785920 (+ 69us) catalog_manager.cc:2758] Locking tablets
0506 19:56:40.854535 (+ 68615us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:40.961825 (+107290us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:40.961914 (+ 89us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:40.961963 (+ 49us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:40.962542 (+ 579us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:40.897618 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:40.897731 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:40.897737 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:40.897740 (+ 3us) tablet.cc:660] Decoding operations
0506 19:56:40.897901 (+ 161us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:40.897919 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:40.897927 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:40.907301 (+ 9374us) tablet.cc:699] Row locks acquired
0506 19:56:40.907305 (+ 4us) write_op.cc:260] PREPARE: finished
0506 19:56:40.907350 (+ 45us) write_op.cc:270] Start()
0506 19:56:40.907386 (+ 36us) write_op.cc:276] Timestamp: P: 1715025400907262 usec, L: 0
0506 19:56:40.907390 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:40.907744 (+ 354us) log.cc:842] Serialized 731 byte log entry
0506 19:56:40.959043 (+ 51299us) op_driver.cc:444] REPLICATION: finished
0506 19:56:40.959823 (+ 780us) write_op.cc:301] APPLY: starting
0506 19:56:40.959841 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:40.959894 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:40.959929 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:40.959998 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:40.960036 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:40.960044 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:40.960129 (+ 85us) log.cc:842] Serialized 29 byte log entry
0506 19:56:40.960193 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.960216 (+ 23us) write_op.cc:454] Released schema lock
0506 19:56:40.960224 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:40.960278 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.960281 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":727,"apply.run_cpu_time_us":511,"apply.run_wall_time_us":511,"num_ops":2,"prepare.queue_time_us":42501,"prepare.run_cpu_time_us":971,"prepare.run_wall_time_us":10953,"raft.queue_time_us":328,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":51619}]]}
I20240506 19:56:40.963219 12986 raft_consensus.cc:2454] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:40.963271 13249 ts_tablet_manager.cc:1367] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.001s
I20240506 19:56:40.963121 13243 rpcz_store.cc:276] 0506 19:56:37.052871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.053073 (+ 202us) service_pool.cc:225] Handling call
0506 19:56:40.928530 (+3875457us) ts_tablet_manager.cc:1120] Deleting tablet da41ef03182445f1946584c2bf015139
0506 19:56:40.928540 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.928670 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3875149}
I20240506 19:56:40.963973 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 274) took 3621 ms. Trace:
I20240506 19:56:40.964023 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c1ff7a3638004012b08c973192c055a5
I20240506 19:56:40.964082 13243 rpcz_store.cc:276] 0506 19:56:37.342864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.342929 (+ 65us) service_pool.cc:225] Handling call
0506 19:56:40.963797 (+3620868us) ts_tablet_manager.cc:1120] Deleting tablet 7d23f0a5ff6d430282256e9bddc186b9
0506 19:56:40.963809 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.963934 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3620435}
I20240506 19:56:40.964156 13249 raft_consensus.cc:348] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.964376 13249 raft_consensus.cc:374] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.964457 13249 raft_consensus.cc:729] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.964485 13243 tablet_replica.cc:330] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:40.964535 12517 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 468) took 1034 ms. Trace:
I20240506 19:56:40.964720 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ff7a3638004012b08c973192c055a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.964759 13243 raft_consensus.cc:2229] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:40.964730 13249 consensus_queue.cc:260] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.964944 13243 raft_consensus.cc:2258] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:40.964612 12517 rpcz_store.cc:276] 0506 19:56:39.930109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.774381 (+844272us) service_pool.cc:225] Handling call
0506 19:56:40.780717 (+ 6336us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:40.790758 (+ 10041us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:40.790845 (+ 87us) catalog_manager.cc:2758] Locking tablets
0506 19:56:40.855284 (+ 64439us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:40.963842 (+108558us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:40.963914 (+ 72us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:40.963959 (+ 45us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:40.964479 (+ 520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:40.908594 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:40.908698 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:40.908704 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:40.908706 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:40.908825 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:40.908834 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:40.908838 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:40.908927 (+ 89us) tablet.cc:699] Row locks acquired
0506 19:56:40.908929 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:40.908958 (+ 29us) write_op.cc:270] Start()
0506 19:56:40.908982 (+ 24us) write_op.cc:276] Timestamp: P: 1715025400908875 usec, L: 0
0506 19:56:40.908985 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:40.909134 (+ 149us) log.cc:842] Serialized 731 byte log entry
0506 19:56:40.959538 (+ 50404us) op_driver.cc:444] REPLICATION: finished
0506 19:56:40.960377 (+ 839us) write_op.cc:301] APPLY: starting
0506 19:56:40.960392 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:40.960431 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:40.960462 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:40.960521 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:40.960548 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:40.960554 (+ 6us) write_op.cc:312] APPLY: finished
0506 19:56:40.960645 (+ 91us) log.cc:842] Serialized 29 byte log entry
0506 19:56:40.960688 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.960704 (+ 16us) write_op.cc:454] Released schema lock
0506 19:56:40.960710 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:40.960755 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:40.960759 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":798,"apply.run_cpu_time_us":426,"apply.run_wall_time_us":425,"num_ops":2,"prepare.queue_time_us":52798,"prepare.run_cpu_time_us":598,"prepare.run_wall_time_us":597,"replication_time_us":50532}]]}
I20240506 19:56:40.965266 13249 ts_tablet_manager.cc:1398] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:40.965590 13249 tablet_bootstrap.cc:492] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:40.965677 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 373e04e93a5543f9934077006eaf2ba9
I20240506 19:56:40.966560 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1a6e0126d9854215a0e174a76c0a0391
I20240506 19:56:40.966995 12515 catalog_manager.cc:6554] ReplaceTablet: tablet 1a6e0126d9854215a0e174a76c0a0391 deleted and replaced by tablet 986f9c83b04f41b49e9b78fce731a90f
I20240506 19:56:40.967288 13249 tablet_bootstrap.cc:654] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.968452 13057 leader_election.cc:302] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:40.968768 13057 raft_consensus.cc:2670] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:40.969781 13243 ts_tablet_manager.cc:1875] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:40.975322 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7d23f0a5ff6d430282256e9bddc186b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d23f0a5ff6d430282256e9bddc186b9 already in progress: creating tablet
I20240506 19:56:40.975950 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.976209 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ff7a3638004012b08c973192c055a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.976377 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e04e93a5543f9934077006eaf2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.976533 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 with delete_type TABLET_DATA_DELETED (replaced by tablet 986f9c83b04f41b49e9b78fce731a90f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:40.977288 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 with delete_type TABLET_DATA_DELETED (replaced by tablet 986f9c83b04f41b49e9b78fce731a90f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.977372 13716 raft_consensus.cc:482] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:40.977540 13716 raft_consensus.cc:504] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.977707 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e04e93a5543f9934077006eaf2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:40.978367 13716 leader_election.cc:288] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:40.978626 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.978833 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ff7a3638004012b08c973192c055a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.979022 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e04e93a5543f9934077006eaf2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.979204 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 with delete_type TABLET_DATA_DELETED (replaced by tablet 986f9c83b04f41b49e9b78fce731a90f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:40.980070 13249 tablet_bootstrap.cc:492] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.980219 13249 ts_tablet_manager.cc:1367] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240506 19:56:40.981181 13249 raft_consensus.cc:348] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.981412 13249 raft_consensus.cc:374] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.981509 13249 raft_consensus.cc:729] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.981467 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8a9ecf47d7474c82758417449d940c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:40.981861 12986 raft_consensus.cc:2379] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef18e88b87064ed4b6c20f9325ebcc33 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:40.981766 13249 consensus_queue.cc:260] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:40.982470 13249 ts_tablet_manager.cc:1398] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:40.982738 13249 tablet_bootstrap.cc:492] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:40.983845 12783 leader_election.cc:341] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 6d8a9ecf47d7474c82758417449d940c
I20240506 19:56:40.984002 12783 leader_election.cc:302] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:40.984337 13249 tablet_bootstrap.cc:654] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.984687 13716 raft_consensus.cc:3037] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:40.988139 13716 raft_consensus.cc:2726] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:40.989344 13244 tablet_bootstrap.cc:492] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:40.989583 13244 ts_tablet_manager.cc:1367] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.005s
I20240506 19:56:40.990458 13244 raft_consensus.cc:348] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.990731 13244 raft_consensus.cc:374] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.990830 13244 raft_consensus.cc:729] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.991114 13244 consensus_queue.cc:260] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.993044 13244 ts_tablet_manager.cc:1398] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:40.993429 13244 tablet_bootstrap.cc:492] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:40.994529 13249 tablet_bootstrap.cc:492] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:40.994755 13249 ts_tablet_manager.cc:1367] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240506 19:56:40.995834 13249 raft_consensus.cc:348] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.996073 13249 raft_consensus.cc:374] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:40.996244 13249 raft_consensus.cc:729] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:40.996537 13249 consensus_queue.cc:260] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:40.997157 13249 ts_tablet_manager.cc:1398] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:40.997509 13249 tablet_bootstrap.cc:492] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:40.999074 13249 tablet_bootstrap.cc:654] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:40.998366 13716 raft_consensus.cc:482] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.001515 13716 raft_consensus.cc:504] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.002213 13716 leader_election.cc:288] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.002418 13244 tablet_bootstrap.cc:654] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.003011 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e3f798d6484d4fabd55f9b4c98a911" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:41.003854 12783 leader_election.cc:341] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: c5e3f798d6484d4fabd55f9b4c98a911
I20240506 19:56:41.004954 12986 raft_consensus.cc:2379] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef18e88b87064ed4b6c20f9325ebcc33 in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:56:41.005565 12781 leader_election.cc:302] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:41.006206 13716 raft_consensus.cc:3037] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:41.007273 13245 tablet_bootstrap.cc:654] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.009984 13716 raft_consensus.cc:2726] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.012681 13242 ts_tablet_manager.cc:1888] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.016341 13246 ts_tablet_manager.cc:1888] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.019433 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.019729 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.019919 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.021421 13243 ts_tablet_manager.cc:1888] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.023010 13242 log.cc:1196] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/92951843c8bd446aa86436b65d9eb793
I20240506 19:56:41.023626 13242 ts_tablet_manager.cc:1909] T 92951843c8bd446aa86436b65d9eb793 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:41.026443 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b803ae289344f88b604a2053881d59b consensus state has no leader
I20240506 19:56:41.027958 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e3807d65b604472889ea0fa7e4b5e69b" } new_table_name: "default.Table_1d1d1064266d4b68934531d7c38d2801" modify_external_catalogs: true
I20240506 19:56:41.028296 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_77ba2f5ca8614d1ebb987a154041901c" } modify_external_catalogs: true
I20240506 19:56:41.028507 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_77ba2f5ca8614d1ebb987a154041901c" } modify_external_catalogs: true
I20240506 19:56:41.029188 12517 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:47760 (request call id 1) took 1047 ms. Trace:
I20240506 19:56:41.022311 13249 tablet_bootstrap.cc:492] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:41.030138 13249 ts_tablet_manager.cc:1367] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.012s
I20240506 19:56:41.030786 12517 rpcz_store.cc:276] 0506 19:56:39.982175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.965219 (+983044us) service_pool.cc:225] Handling call
0506 19:56:41.029023 (+ 63804us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x616000021d54. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 00007f8feed4915d 00007f8feed49253 00007f8feba7ad41 00007f8ff366734d 00007f8ff37fc1bf 00007f8fea7f7227 00007f8fea7f6f55 00007f8fe6f1104e 00007f8fe6f0ff2e 00007f8fe6f16405 00007f8fe6f16d2f
0506 19:56:41.029125 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146080000}
I20240506 19:56:41.031167 13249 raft_consensus.cc:348] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.031379 13249 raft_consensus.cc:374] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.031471 13249 raft_consensus.cc:729] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.031689 13249 consensus_queue.cc:260] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.032402 13249 ts_tablet_manager.cc:1398] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:41.029119 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8a9ecf47d7474c82758417449d940c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.033208 13243 log.cc:1196] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/4a080ccac6394210b0b24b65843f9d75
I20240506 19:56:41.034824 13243 ts_tablet_manager.cc:1909] T 4a080ccac6394210b0b24b65843f9d75 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.035427 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 292) took 3371 ms. Trace:
I20240506 19:56:41.035037 13249 tablet_bootstrap.cc:492] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:41.036592 13242 rpcz_store.cc:276] 0506 19:56:37.664321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.664363 (+ 42us) service_pool.cc:225] Handling call
0506 19:56:40.950792 (+3286429us) ts_tablet_manager.cc:1120] Deleting tablet 92951843c8bd446aa86436b65d9eb793
0506 19:56:40.950803 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.966797 (+ 15994us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.012537 (+ 45740us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.035370 (+ 22833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3286259}
I20240506 19:56:41.026929 13245 tablet_bootstrap.cc:492] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:41.027540 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 1667a0bbd3af4d678434b068c47cc518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.038288 13245 ts_tablet_manager.cc:1367] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.004s
I20240506 19:56:41.039232 13245 raft_consensus.cc:348] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.039500 13245 raft_consensus.cc:374] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.039610 13245 raft_consensus.cc:729] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.039882 13245 consensus_queue.cc:260] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.040294 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91888e88234cbdbd00f48dc1574653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.041961 13245 ts_tablet_manager.cc:1398] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:41.042300 13245 tablet_bootstrap.cc:492] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:41.043071 13246 log.cc:1196] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:41.037890 13244 tablet_bootstrap.cc:492] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:41.043964 13244 ts_tablet_manager.cc:1367] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.002s
I20240506 19:56:41.045015 13244 raft_consensus.cc:348] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.045372 13244 raft_consensus.cc:374] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.045517 13244 raft_consensus.cc:729] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.045871 13244 consensus_queue.cc:260] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.049952 13244 ts_tablet_manager.cc:1398] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:41.050555 13244 tablet_bootstrap.cc:492] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:41.050908 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 273) took 3693 ms. Trace:
I20240506 19:56:41.051296 13246 ts_tablet_manager.cc:1909] T 72a3bed0eb234e21b890242e9d23a5b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.051430 13243 rpcz_store.cc:276] 0506 19:56:37.342714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.343422 (+ 708us) service_pool.cc:225] Handling call
0506 19:56:40.964406 (+3620984us) ts_tablet_manager.cc:1120] Deleting tablet 4a080ccac6394210b0b24b65843f9d75
0506 19:56:40.964416 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:40.973792 (+ 9376us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.021230 (+ 47438us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.036445 (+ 15215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3620790}
I20240506 19:56:41.053540 13244 tablet_bootstrap.cc:654] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.055850 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_da2ec44bc10845a7a4fbf93f4916c652"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.056053 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dfcabd286a794ef19121c86363912db0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.056672 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da2ec44bc10845a7a4fbf93f4916c652 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.055881 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8bb7368b19d441baa65305aa9e64514d" } new_table_name: "default.Table_978baf79e99f4eb8840e5030d3f7b93c" modify_external_catalogs: true
I20240506 19:56:41.058349 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dbebd827d8ed405bae594e5adc6c14a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.061502 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbebd827d8ed405bae594e5adc6c14a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.065243 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 92951843c8bd446aa86436b65d9eb793 (table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]) successfully deleted
I20240506 19:56:41.065464 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cf03515d64254430b39e443cedd953f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.066260 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfcabd286a794ef19121c86363912db0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:41.066992 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf03515d64254430b39e443cedd953f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:41.070348 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63f56d2180b1472a8168fb3a63589065 consensus state has no leader
I20240506 19:56:41.070881 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea42f8ad6dc0468fba4b4876d9651595" } new_table_name: "default.Table_4897151eca6e4c93be0340b5da47fd8c" modify_external_catalogs: true
I20240506 19:56:41.053869 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet b38453bf678748399339806d12993c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.054100 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe with delete_type TABLET_DATA_DELETED (replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.073762 13249 tablet_bootstrap.cc:654] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.074082 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 272) took 3731 ms. Trace:
I20240506 19:56:41.074278 13243 rpcz_store.cc:276] 0506 19:56:37.342587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.343644 (+ 1057us) service_pool.cc:225] Handling call
0506 19:56:41.073788 (+3730144us) ts_tablet_manager.cc:1120] Deleting tablet a8b3ed0b56aa478abb44347afca8fcf6
0506 19:56:41.073800 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.074003 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3729969}
I20240506 19:56:41.072638 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 4a080ccac6394210b0b24b65843f9d75 (table default.Table_bb529dd8d37f40449f27b0e07cbc93dc [id=4b9a2f4ea4b1460f89e8981e952e0d2d]) successfully deleted
W20240506 19:56:41.075176 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a8b3ed0b56aa478abb44347afca8fcf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8b3ed0b56aa478abb44347afca8fcf6 already in progress: creating tablet
I20240506 19:56:41.073561 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_960b875a518745a38b1a1e6699ec3a2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.075825 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_960b875a518745a38b1a1e6699ec3a2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.054692 13245 tablet_bootstrap.cc:654] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.077630 13246 tablet_replica.cc:330] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:41.077934 13246 raft_consensus.cc:2229] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:41.078148 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7b803ae289344f88b604a2053881d59b (table default.Table_2f9d2a9b611448949d0a7fc4f5146078 [id=d2ffd5634f2042b7a85cbe74d6ca1df5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b803ae289344f88b604a2053881d59b consensus state has no leader
I20240506 19:56:41.078445 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 290) took 3563 ms. Trace:
I20240506 19:56:41.078562 13243 rpcz_store.cc:276] 0506 19:56:37.511031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.511102 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:41.074762 (+3563660us) ts_tablet_manager.cc:1120] Deleting tablet 2f7f7171d06948388bc0fdb3514202ed
0506 19:56:41.074772 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.074863 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3563342}
I20240506 19:56:41.078734 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 72a3bed0eb234e21b890242e9d23a5b4 (table default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]) successfully deleted
I20240506 19:56:41.064069 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 393073bca2a04d15b3e583700262f3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b529bb9362924caca399491e7d9e03cd) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.079794 13246 raft_consensus.cc:2258] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.076431 13242 tablet_replica.cc:330] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:41.082346 13242 raft_consensus.cc:2229] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:41.082475 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2f7f7171d06948388bc0fdb3514202ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7f7171d06948388bc0fdb3514202ed already in progress: creating tablet
I20240506 19:56:41.082662 13242 raft_consensus.cc:2258] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.082795 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 291) took 3571 ms. Trace:
I20240506 19:56:41.084122 13242 ts_tablet_manager.cc:1875] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.084218 13246 ts_tablet_manager.cc:1875] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.085276 13243 rpcz_store.cc:276] 0506 19:56:37.511169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.511401 (+ 232us) service_pool.cc:225] Handling call
0506 19:56:41.082616 (+3571215us) ts_tablet_manager.cc:1120] Deleting tablet b899fb38bfed4e02b3708f6abec80e0a
0506 19:56:41.082627 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.082738 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3571003}
I20240506 19:56:41.089861 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 292) took 3578 ms. Trace:
I20240506 19:56:41.091228 13243 rpcz_store.cc:276] 0506 19:56:37.511347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.511647 (+ 300us) service_pool.cc:225] Handling call
0506 19:56:41.089633 (+3577986us) ts_tablet_manager.cc:1120] Deleting tablet 472c63d8995346d78633154a8f0d4c2b
0506 19:56:41.089648 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.089806 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3577701}
I20240506 19:56:41.093392 13243 tablet_replica.cc:330] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.093694 13243 raft_consensus.cc:2229] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:41.093987 13245 tablet_bootstrap.cc:492] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:41.094118 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b899fb38bfed4e02b3708f6abec80e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b899fb38bfed4e02b3708f6abec80e0a already in progress: creating tablet
I20240506 19:56:41.094331 13245 ts_tablet_manager.cc:1367] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
W20240506 19:56:41.094460 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 472c63d8995346d78633154a8f0d4c2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472c63d8995346d78633154a8f0d4c2b already in progress: creating tablet
I20240506 19:56:41.094720 13243 pending_rounds.cc:62] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:41.095377 13245 raft_consensus.cc:348] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.095638 13245 raft_consensus.cc:374] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.095736 13245 raft_consensus.cc:729] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.096028 13245 consensus_queue.cc:260] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.096657 13245 ts_tablet_manager.cc:1398] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:41.096895 13243 pending_rounds.cc:68] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744025109397504 op_type: NO_OP noop_request { }
I20240506 19:56:41.097100 13243 raft_consensus.cc:2866] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:41.097290 13243 raft_consensus.cc:2258] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.100395 13243 ts_tablet_manager.cc:1875] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:41.103929 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3a8531a1793404699532295164a3895 consensus state has no leader
I20240506 19:56:41.104456 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9d643b8fae2443eab31564f9f0440d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.105197 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d643b8fae2443eab31564f9f0440d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.106716 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aabc53fdc37a4ffc89e4fe789817f8f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.107385 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aabc53fdc37a4ffc89e4fe789817f8f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.110466 13246 ts_tablet_manager.cc:1888] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.115958 13249 tablet_bootstrap.cc:492] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:41.116147 13249 ts_tablet_manager.cc:1367] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.001s
I20240506 19:56:41.117332 13249 raft_consensus.cc:348] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.117568 13249 raft_consensus.cc:374] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.117640 13249 raft_consensus.cc:729] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.117889 13249 consensus_queue.cc:260] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.120381 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_83626d3a5f3348679ec14bda1af29deb" } new_table_name: "default.Table_268439c35972405584472829b94a5011" modify_external_catalogs: true
I20240506 19:56:41.122898 13249 ts_tablet_manager.cc:1398] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240506 19:56:41.125761 13719 raft_consensus.cc:482] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.134894 13719 raft_consensus.cc:504] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:41.135730 13719 leader_election.cc:288] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.123378 13249 tablet_bootstrap.cc:492] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:41.137056 13243 ts_tablet_manager.cc:1888] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:41.137188 13243 log.cc:1196] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/8842523a97954e3f81ffd87530201d93
I20240506 19:56:41.137662 13243 ts_tablet_manager.cc:1909] T 8842523a97954e3f81ffd87530201d93 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.138186 13246 log.cc:1196] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:41.138409 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8bea4c0b88b344d6ba5cf80917534deb" } modify_external_catalogs: true
I20240506 19:56:41.138537 13246 ts_tablet_manager.cc:1909] T 40ece0ad4df247eb9d380ed1c755ac99 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.138618 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8bea4c0b88b344d6ba5cf80917534deb" } modify_external_catalogs: true
I20240506 19:56:41.114660 13245 tablet_bootstrap.cc:492] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:41.139158 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38453bf678748399339806d12993c88" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:41.139386 13122 raft_consensus.cc:2454] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:41.140026 12922 leader_election.cc:302] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:41.140295 13719 raft_consensus.cc:2781] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:41.140406 13719 raft_consensus.cc:482] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:41.140509 13719 raft_consensus.cc:3037] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:41.140966 13245 tablet_bootstrap.cc:654] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.141467 13246 tablet_replica.cc:330] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:41.141731 13246 raft_consensus.cc:2229] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.141991 13246 raft_consensus.cc:2258] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.143960 13719 raft_consensus.cc:504] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:41.146649 13719 leader_election.cc:288] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.147271 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38453bf678748399339806d12993c88" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:41.147974 12922 leader_election.cc:341] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.148378 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_503cbc59b232469d96f78bbec69b3d57 [id=f23583e607a4419986d06a38c45bbdf5]). Expected version 1 got 0
I20240506 19:56:41.149035 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 40ece0ad4df247eb9d380ed1c755ac99 (table default.Table_d91e484ef0cc4adb86af57d549a7e641 [id=9ad0be2302b94351876dba7a918a8d22]) successfully deleted
W20240506 19:56:41.149375 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4630abf54f024da5be5b45efccc0051d consensus state has no leader
I20240506 19:56:41.149852 13090 tablet_service.cc:1467] Processing CreateTablet for tablet ecc88443c04b42d19cc7fba339eeeaf5 (DEFAULT_TABLE table=default.Table_fa840bd001a04582ac72848588e27126 [id=6f385f8f969245adbb034e18894b5df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.150280 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc88443c04b42d19cc7fba339eeeaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.151278 13246 ts_tablet_manager.cc:1875] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.153676 13249 tablet_bootstrap.cc:654] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.153818 13102 tablet_service.cc:1467] Processing CreateTablet for tablet b625c2bbd89746bc90b8c4c2483b3741 (DEFAULT_TABLE table=default.Table_a62a1c7b556341a88bc49f8caa3c8238 [id=862ab2f52d2c472eba72311e18532f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.154253 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b625c2bbd89746bc90b8c4c2483b3741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.154441 12967 tablet_service.cc:1467] Processing CreateTablet for tablet ecc88443c04b42d19cc7fba339eeeaf5 (DEFAULT_TABLE table=default.Table_fa840bd001a04582ac72848588e27126 [id=6f385f8f969245adbb034e18894b5df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.154798 13245 tablet_bootstrap.cc:492] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:41.154834 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc88443c04b42d19cc7fba339eeeaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.154949 13245 ts_tablet_manager.cc:1367] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.005s
I20240506 19:56:41.154999 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 276) took 3703 ms. Trace:
I20240506 19:56:41.155100 13243 rpcz_store.cc:276] 0506 19:56:37.451165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.451221 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:41.093267 (+3642046us) ts_tablet_manager.cc:1120] Deleting tablet 8842523a97954e3f81ffd87530201d93
0506 19:56:41.093279 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.107824 (+ 14545us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.136935 (+ 29111us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.154938 (+ 18003us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11957,"tablet-delete.queue_time_us":3578380}
I20240506 19:56:41.155490 13243 tablet_replica.cc:330] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.155692 13243 raft_consensus.cc:2229] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 2 LEADER]: Raft consensus shutting down.
I20240506 19:56:41.155959 13243 pending_rounds.cc:62] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:41.155838 13245 raft_consensus.cc:348] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.156055 13245 raft_consensus.cc:374] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.156023 13243 pending_rounds.cc:68] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7024744041480765440 op_type: NO_OP noop_request { }
I20240506 19:56:41.156143 13245 raft_consensus.cc:729] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.156167 13243 raft_consensus.cc:2866] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:41.156270 13243 raft_consensus.cc:2258] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.156961 13243 ts_tablet_manager.cc:1875] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.156358 13245 consensus_queue.cc:260] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:41.157397 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da06187c83b94d03ae4ebb67a2873cf0 consensus state has no leader
I20240506 19:56:41.158231 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet a1160ab972124630a9be7abeefb4aa56 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:41.160084 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet ee59a76d52fc48d7bcf3256c359d3c02 (table default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7 [id=d9b546ebb0c146a7920cc05c08a29e32]). Expected version 1 got 0
I20240506 19:56:41.163798 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 8842523a97954e3f81ffd87530201d93 (table default.Table_0594be752cc441b1af65142e6d925488 [id=b1277bb384294e7ea3cd5d0a52b5c813]) successfully deleted
I20240506 19:56:41.163906 12961 tablet_service.cc:1467] Processing CreateTablet for tablet b625c2bbd89746bc90b8c4c2483b3741 (DEFAULT_TABLE table=default.Table_a62a1c7b556341a88bc49f8caa3c8238 [id=862ab2f52d2c472eba72311e18532f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.164350 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b625c2bbd89746bc90b8c4c2483b3741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.164930 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.165189 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe with delete_type TABLET_DATA_DELETED (replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.169083 13243 ts_tablet_manager.cc:1888] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240506 19:56:41.169054 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4cbe3de6b961432fbeacafa9ae5fe50c" } modify_external_catalogs: true
I20240506 19:56:41.169271 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4cbe3de6b961432fbeacafa9ae5fe50c" } modify_external_catalogs: true
I20240506 19:56:41.169577 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 8d09eafa82a0423a8358958e78f728e2 (DEFAULT_TABLE table=default.Table_8bea4c0b88b344d6ba5cf80917534deb [id=7968470a4da549f49e951f3d5ad65c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.169881 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 (DEFAULT_TABLE table=default.Table_604f6c5cb5214c97b981184d3202d345 [id=c2d71ff2a3ef40ba997ae05af25bfca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.170017 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d09eafa82a0423a8358958e78f728e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.170240 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f28c1f55fd14aca86cb00e43b3a6b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.170689 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 (DEFAULT_TABLE table=default.Table_604f6c5cb5214c97b981184d3202d345 [id=c2d71ff2a3ef40ba997ae05af25bfca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.170739 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7b7dd900dc5b478280237dfd98e0f303" } modify_external_catalogs: true
I20240506 19:56:41.170941 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7b7dd900dc5b478280237dfd98e0f303" } modify_external_catalogs: true
I20240506 19:56:41.171068 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f28c1f55fd14aca86cb00e43b3a6b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.172840 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 85365758f6a64536ab61e8349de63dea (DEFAULT_TABLE table=default.Table_86f15af99695455a9f33070a0bc8563f [id=c266c557aaf24c0d9eaba0cc04b6988f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.173218 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85365758f6a64536ab61e8349de63dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.173563 12957 tablet_service.cc:1467] Processing CreateTablet for tablet cd6245e4adce4038ad6f988877d75045 (DEFAULT_TABLE table=default.Table_2486dda60f0b4f2b9a63dbf17885bc5e [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.173604 13085 tablet_service.cc:1467] Processing CreateTablet for tablet cd6245e4adce4038ad6f988877d75045 (DEFAULT_TABLE table=default.Table_2486dda60f0b4f2b9a63dbf17885bc5e [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.173977 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6245e4adce4038ad6f988877d75045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.174010 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6245e4adce4038ad6f988877d75045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.177665 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 85365758f6a64536ab61e8349de63dea (DEFAULT_TABLE table=default.Table_86f15af99695455a9f33070a0bc8563f [id=c266c557aaf24c0d9eaba0cc04b6988f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.178068 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85365758f6a64536ab61e8349de63dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.179736 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_503cbc59b232469d96f78bbec69b3d57 [id=f23583e607a4419986d06a38c45bbdf5]). Expected version 1 got 0
I20240506 19:56:41.180473 13243 log.cc:1196] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/92951843c8bd446aa86436b65d9eb793
I20240506 19:56:41.180907 13243 ts_tablet_manager.cc:1909] T 92951843c8bd446aa86436b65d9eb793 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.181674 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 8d09eafa82a0423a8358958e78f728e2 (DEFAULT_TABLE table=default.Table_8bea4c0b88b344d6ba5cf80917534deb [id=7968470a4da549f49e951f3d5ad65c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.182165 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d09eafa82a0423a8358958e78f728e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.182883 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 277) took 3731 ms. Trace:
I20240506 19:56:41.182994 13243 rpcz_store.cc:276] 0506 19:56:37.451287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.451343 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:41.155395 (+3704052us) ts_tablet_manager.cc:1120] Deleting tablet 92951843c8bd446aa86436b65d9eb793
0506 19:56:41.155403 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.162215 (+ 6812us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.168959 (+ 6744us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.182820 (+ 13861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3640313}
I20240506 19:56:41.183398 13243 tablet_replica.cc:330] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.183616 13243 raft_consensus.cc:2229] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.185487 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 92951843c8bd446aa86436b65d9eb793 (table default.Table_d8c0cbd18a09476cb0895d1ce1c89de9 [id=80f1c0ce8819489db4917d1d5a13a54c]) successfully deleted
I20240506 19:56:41.185554 13243 raft_consensus.cc:2258] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.185989 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b562516d9b845789bf9d901c015e37c" } modify_external_catalogs: true
I20240506 19:56:41.186214 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b562516d9b845789bf9d901c015e37c" } modify_external_catalogs: true
I20240506 19:56:41.186815 13243 ts_tablet_manager.cc:1875] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.214148 13243 ts_tablet_manager.cc:1888] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:41.214354 13243 log.cc:1196] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/03e896b3ba7c4864a26cdfce76fadcfb
I20240506 19:56:41.215978 13243 ts_tablet_manager.cc:1909] T 03e896b3ba7c4864a26cdfce76fadcfb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.217581 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9bca56a380f341c28cf5c3f683024098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.218277 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bca56a380f341c28cf5c3f683024098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.222461 13102 tablet_service.cc:1467] Processing CreateTablet for tablet e63baf59dc8244daad831b2987a51cfb (DEFAULT_TABLE table=default.Table_25774424d24c4cd68a59fd89cae9b492 [id=3dcb08c9175142c7831f2224f317a453]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.223304 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63baf59dc8244daad831b2987a51cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.220715 12961 tablet_service.cc:1467] Processing CreateTablet for tablet e63baf59dc8244daad831b2987a51cfb (DEFAULT_TABLE table=default.Table_25774424d24c4cd68a59fd89cae9b492 [id=3dcb08c9175142c7831f2224f317a453]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.225081 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63baf59dc8244daad831b2987a51cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.228574 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_35348a0401e440239d4fa0424c80bd0f" } new_table_name: "default.Table_384bed2d6d984f9d88c62381e38b6321" modify_external_catalogs: true
I20240506 19:56:41.228751 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_57c82ed0d1044d20b72a0e9341b299c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.229378 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57c82ed0d1044d20b72a0e9341b299c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.232049 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d6e54543d1544b1c8cff3e890e787337
I20240506 19:56:41.232928 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a1160ab972124630a9be7abeefb4aa56
I20240506 19:56:41.233387 12516 catalog_manager.cc:6554] ReplaceTablet: tablet a1160ab972124630a9be7abeefb4aa56 deleted and replaced by tablet 44ce8749846847108edf5f554139cbd9
I20240506 19:56:41.233513 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e54543d1544b1c8cff3e890e787337 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.233736 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8d09eafa82a0423a8358958e78f728e2
I20240506 19:56:41.234391 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 234e4f3439e84b5fbb2f7bab1dbd0822
I20240506 19:56:41.235066 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:56:41.235189 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d09eafa82a0423a8358958e78f728e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.235460 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet a1160ab972124630a9be7abeefb4aa56 with delete_type TABLET_DATA_DELETED (replaced by tablet 44ce8749846847108edf5f554139cbd9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.236614 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.236673 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet a1160ab972124630a9be7abeefb4aa56 with delete_type TABLET_DATA_DELETED (replaced by tablet 44ce8749846847108edf5f554139cbd9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.236886 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.236953 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d09eafa82a0423a8358958e78f728e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.237166 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.237409 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.238240 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e54543d1544b1c8cff3e890e787337 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.242043 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 275) took 3791 ms. Trace:
I20240506 19:56:41.242291 13243 rpcz_store.cc:276] 0506 19:56:37.450738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.450837 (+ 99us) service_pool.cc:225] Handling call
0506 19:56:41.183306 (+3732469us) ts_tablet_manager.cc:1120] Deleting tablet 03e896b3ba7c4864a26cdfce76fadcfb
0506 19:56:41.183321 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.209141 (+ 25820us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.213958 (+ 4817us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.241978 (+ 28020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3621428}
I20240506 19:56:41.243036 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a9e90b93023e40b3bafbddf7bbcc10b2
I20240506 19:56:41.243244 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 294) took 3614 ms. Trace:
I20240506 19:56:41.244007 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.244521 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 03e896b3ba7c4864a26cdfce76fadcfb (table default.Table_fe77d7d403a041b4b9f6a1c97177f461 [id=15e39c71182c4a089e9c950679153305]) successfully deleted
I20240506 19:56:41.244910 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.245776 13243 rpcz_store.cc:276] 0506 19:56:37.628470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.628535 (+ 65us) service_pool.cc:225] Handling call
0506 19:56:41.243022 (+3614487us) ts_tablet_manager.cc:1120] Deleting tablet d05c5ca7ff864fa98f3494032729e978
0506 19:56:41.243036 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.243198 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3614173}
I20240506 19:56:41.246559 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 297) took 3574 ms. Trace:
I20240506 19:56:41.252033 13243 rpcz_store.cc:276] 0506 19:56:37.672432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.672496 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:41.246367 (+3573871us) ts_tablet_manager.cc:1120] Deleting tablet 5988b89cb52e4f858b52862138bed025
0506 19:56:41.246381 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.246508 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573612}
I20240506 19:56:41.253149 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 298) took 3580 ms. Trace:
I20240506 19:56:41.253419 13243 rpcz_store.cc:276] 0506 19:56:37.672584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.672816 (+ 232us) service_pool.cc:225] Handling call
0506 19:56:41.252936 (+3580120us) ts_tablet_manager.cc:1120] Deleting tablet 4acde41b2c03440a89243aae008779e1
0506 19:56:41.252952 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.253096 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3579911}
W20240506 19:56:41.253891 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5988b89cb52e4f858b52862138bed025 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5988b89cb52e4f858b52862138bed025 already in progress: creating tablet
I20240506 19:56:41.254173 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 299) took 3580 ms. Trace:
W20240506 19:56:41.254419 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4acde41b2c03440a89243aae008779e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4acde41b2c03440a89243aae008779e1 already in progress: creating tablet
I20240506 19:56:41.254441 13243 rpcz_store.cc:276] 0506 19:56:37.673828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.673886 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:41.253994 (+3580108us) ts_tablet_manager.cc:1120] Deleting tablet 517c09e3bf7046ccba6ed25b08e3460e
0506 19:56:41.254007 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.254125 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3579904}
I20240506 19:56:41.255067 13243 tablet_replica.cc:330] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.255470 13243 raft_consensus.cc:2229] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:41.255998 13243 raft_consensus.cc:2258] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:41.256350 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 517c09e3bf7046ccba6ed25b08e3460e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 517c09e3bf7046ccba6ed25b08e3460e already in progress: creating tablet
I20240506 19:56:41.256904 13243 ts_tablet_manager.cc:1875] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:41.264642 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2a5a5e193f245de96cc82986e292f05 consensus state has no leader
I20240506 19:56:41.268080 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a0fc2d32e27a4634bece00d1d8abd75c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.268877 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0fc2d32e27a4634bece00d1d8abd75c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.271993 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bd4b7877d06746518f96f0601873f10f" } new_table_name: "default.Table_7b7fe87d3acd4184b95b2dd1026980d5" modify_external_catalogs: true
I20240506 19:56:41.275379 13243 ts_tablet_manager.cc:1888] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:41.275712 13243 log.cc:1196] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1b7f56fea34946e590dae0621b28f9c6
I20240506 19:56:41.276400 13243 ts_tablet_manager.cc:1909] T 1b7f56fea34946e590dae0621b28f9c6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.278355 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 302) took 3568 ms. Trace:
I20240506 19:56:41.279510 13243 rpcz_store.cc:276] 0506 19:56:37.709759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.709821 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:41.254971 (+3545150us) ts_tablet_manager.cc:1120] Deleting tablet 1b7f56fea34946e590dae0621b28f9c6
0506 19:56:41.254983 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.264737 (+ 9754us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.275190 (+ 10453us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.278264 (+ 3074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3544909}
I20240506 19:56:41.280495 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1b7f56fea34946e590dae0621b28f9c6 (table default.Table_a2fb551b2fc541e38f0a655c4539060b [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:41.280581 13243 tablet_replica.cc:330] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.280866 13243 raft_consensus.cc:2229] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.281558 13243 raft_consensus.cc:2258] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.173625 13245 ts_tablet_manager.cc:1398] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240506 19:56:41.283797 13243 ts_tablet_manager.cc:1875] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.286383 13245 tablet_bootstrap.cc:492] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:41.287350 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fed2f12eda54be0879b1e9c33ca3e31 has no consensus state
I20240506 19:56:41.288071 13245 tablet_bootstrap.cc:654] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.314898 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_46aaa890b4ce4d02919bab7b358c7a79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.315274 13246 ts_tablet_manager.cc:1888] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.315426 13246 log.cc:1196] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b38453bf678748399339806d12993c88
W20240506 19:56:41.315519 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46aaa890b4ce4d02919bab7b358c7a79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.315821 13246 ts_tablet_manager.cc:1909] T b38453bf678748399339806d12993c88 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.319684 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b38453bf678748399339806d12993c88 (table default.Table_62612e9478664e9d86a80129badd6c12 [id=8f671c29345e43b6ba8379f8182cba65]) successfully deleted
I20240506 19:56:41.322467 13719 consensus_queue.cc:1035] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:41.322562 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_503cbc59b232469d96f78bbec69b3d57" } new_table_name: "default.Table_bcd38427a79c48d1badd4f2536ff238f" modify_external_catalogs: true
I20240506 19:56:41.324124 13246 tablet_replica.cc:330] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:41.324362 13246 raft_consensus.cc:2229] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.324558 13246 raft_consensus.cc:2258] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.326453 13243 ts_tablet_manager.cc:1888] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.327490 13730 raft_consensus.cc:482] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.327643 13730 raft_consensus.cc:504] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.328112 13245 tablet_bootstrap.cc:492] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:41.328338 13730 leader_election.cc:288] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.328444 13245 ts_tablet_manager.cc:1367] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.000s
I20240506 19:56:41.329197 13245 raft_consensus.cc:348] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.329972 13245 raft_consensus.cc:374] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.330070 13245 raft_consensus.cc:729] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.330415 13245 consensus_queue.cc:260] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.331223 13245 ts_tablet_manager.cc:1398] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:41.330675 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c656e16c1045dea79bc23b64fdd5e8" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.331524 13245 tablet_bootstrap.cc:492] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:41.331815 13243 log.cc:1196] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/c5e3f798d6484d4fabd55f9b4c98a911
I20240506 19:56:41.331642 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8507072bb8ea467982eab99c978a7ac8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.332023 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_90d5ee072fbd4121ae52144e32010742"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.332500 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8507072bb8ea467982eab99c978a7ac8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.332746 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_df0ba23556a64063af810291bf81830e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.333281 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df0ba23556a64063af810291bf81830e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.333786 13243 ts_tablet_manager.cc:1909] T c5e3f798d6484d4fabd55f9b4c98a911 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:41.334441 13057 leader_election.cc:341] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.334599 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 304) took 3624 ms. Trace:
I20240506 19:56:41.335233 13246 ts_tablet_manager.cc:1875] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:41.332500 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90d5ee072fbd4121ae52144e32010742 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.336335 13245 tablet_bootstrap.cc:654] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.336941 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f7049cb5beca4e30b1f0a315789a1e1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.337572 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7049cb5beca4e30b1f0a315789a1e1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.337622 13243 rpcz_store.cc:276] 0506 19:56:37.710136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.711996 (+ 1860us) service_pool.cc:225] Handling call
0506 19:56:41.280273 (+3568277us) ts_tablet_manager.cc:1120] Deleting tablet c5e3f798d6484d4fabd55f9b4c98a911
0506 19:56:41.280288 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.287346 (+ 7058us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.326303 (+ 38957us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.334556 (+ 8253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3568028}
I20240506 19:56:41.339105 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet c5e3f798d6484d4fabd55f9b4c98a911 (table default.Table_d08a9f7decde4bb69b28102c4b21a5bd [id=249eebb1852745929487a8fff7406ca1]) successfully deleted
I20240506 19:56:41.339845 13243 tablet_replica.cc:330] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.340646 13243 raft_consensus.cc:2229] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.340914 13243 raft_consensus.cc:2258] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:41.340992 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa consensus state has no leader
I20240506 19:56:41.343621 13243 ts_tablet_manager.cc:1875] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.346177 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38453bf678748399339806d12993c88" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.348611 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 2502b423bd1040539fb1a3ebbece3c1c with delete_type TABLET_DATA_DELETED (replaced by tablet 00de09ddcdd948a6a3500a6c9d814c14) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.349025 12813 tablet_service.cc:1467] Processing CreateTablet for tablet ecc88443c04b42d19cc7fba339eeeaf5 (DEFAULT_TABLE table=default.Table_fa840bd001a04582ac72848588e27126 [id=6f385f8f969245adbb034e18894b5df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.350188 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c4128d96524a4c20b2f9a7ceedccfe96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.350613 12821 tablet_service.cc:1467] Processing CreateTablet for tablet b625c2bbd89746bc90b8c4c2483b3741 (DEFAULT_TABLE table=default.Table_a62a1c7b556341a88bc49f8caa3c8238 [id=862ab2f52d2c472eba72311e18532f1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:41.350703 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4128d96524a4c20b2f9a7ceedccfe96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.350683 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c656e16c1045dea79bc23b64fdd5e8" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.351135 13244 tablet_bootstrap.cc:492] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:41.351344 13243 ts_tablet_manager.cc:1888] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.351480 13243 log.cc:1196] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/5516e8aad4e8474692b5b6c40bc483c1
I20240506 19:56:41.351514 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 8d09eafa82a0423a8358958e78f728e2 (DEFAULT_TABLE table=default.Table_8bea4c0b88b344d6ba5cf80917534deb [id=7968470a4da549f49e951f3d5ad65c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.351861 13243 ts_tablet_manager.cc:1909] T 5516e8aad4e8474692b5b6c40bc483c1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.351979 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_db02a40f1e43425ea3ed5c28eaba58b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.352444 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db02a40f1e43425ea3ed5c28eaba58b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.352631 13244 ts_tablet_manager.cc:1367] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.301s user 0.005s sys 0.000s
I20240506 19:56:41.352787 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 303) took 3642 ms. Trace:
I20240506 19:56:41.353164 13243 rpcz_store.cc:276] 0506 19:56:37.709959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.719295 (+ 9336us) service_pool.cc:225] Handling call
0506 19:56:41.338954 (+3619659us) ts_tablet_manager.cc:1120] Deleting tablet 5516e8aad4e8474692b5b6c40bc483c1
0506 19:56:41.338965 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.347629 (+ 8664us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.351230 (+ 3601us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.352731 (+ 1501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3619356}
I20240506 19:56:41.353646 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 308) took 3607 ms. Trace:
I20240506 19:56:41.353678 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 5516e8aad4e8474692b5b6c40bc483c1 (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
I20240506 19:56:41.353825 13243 rpcz_store.cc:276] 0506 19:56:37.746407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.747743 (+ 1336us) service_pool.cc:225] Handling call
0506 19:56:41.353482 (+3605739us) ts_tablet_manager.cc:1120] Deleting tablet bafd3c4706f74e7da36a8f3538760f43
0506 19:56:41.353492 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.353592 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605437}
W20240506 19:56:41.354207 13058 leader_election.cc:341] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.354246 13243 tablet_replica.cc:330] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.354341 13058 leader_election.cc:302] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:41.354367 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bafd3c4706f74e7da36a8f3538760f43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bafd3c4706f74e7da36a8f3538760f43 already in progress: creating tablet
I20240506 19:56:41.354470 13243 raft_consensus.cc:2229] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.354683 13243 raft_consensus.cc:2258] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.355418 13244 raft_consensus.cc:348] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.355710 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fca8db6133de4be59cb24acf084124ef" } new_table_name: "default.Table_5ef3ebc12c1f47278ef201cf63e5e39d" modify_external_catalogs: true
I20240506 19:56:41.357209 13730 raft_consensus.cc:2726] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.357503 13244 raft_consensus.cc:374] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.357605 13244 raft_consensus.cc:729] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.357975 13246 ts_tablet_manager.cc:1888] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:41.358106 13246 log.cc:1196] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/5dc0ac37f67942fc8a31d8d43c839246
W20240506 19:56:41.358404 12921 leader_election.cc:341] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.358826 13246 ts_tablet_manager.cc:1909] T 5dc0ac37f67942fc8a31d8d43c839246 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.359303 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a0fc2d32e27a4634bece00d1d8abd75c" } new_table_name: "default.Table_51c8c97056294104afce55e84e26a550" modify_external_catalogs: true
I20240506 19:56:41.359938 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:41.360534 13246 tablet_replica.cc:330] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:41.360534 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8ffb88e0a0c4125b04f4d8d328da3f5 already in progress: creating tablet
W20240506 19:56:41.360749 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 034d6309de0740d0a5806b3847e334cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 034d6309de0740d0a5806b3847e334cb already in progress: creating tablet
I20240506 19:56:41.360757 13246 raft_consensus.cc:2229] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.360918 13246 raft_consensus.cc:2258] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:41.360917 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bece304e0ba048ecbb2d92aa71fdf707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bece304e0ba048ecbb2d92aa71fdf707 already in progress: creating tablet
W20240506 19:56:41.361135 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c1ff7a3638004012b08c973192c055a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1ff7a3638004012b08c973192c055a5 already in progress: creating tablet
I20240506 19:56:41.361006 13244 consensus_queue.cc:260] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:41.361380 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1a6e0126d9854215a0e174a76c0a0391 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a6e0126d9854215a0e174a76c0a0391 already in progress: creating tablet
I20240506 19:56:41.361665 13244 ts_tablet_manager.cc:1398] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:56:41.361724 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 85365758f6a64536ab61e8349de63dea (DEFAULT_TABLE table=default.Table_86f15af99695455a9f33070a0bc8563f [id=c266c557aaf24c0d9eaba0cc04b6988f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:41.362071 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 373e04e93a5543f9934077006eaf2ba9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 373e04e93a5543f9934077006eaf2ba9 already in progress: creating tablet
I20240506 19:56:41.363381 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 (DEFAULT_TABLE table=default.Table_604f6c5cb5214c97b981184d3202d345 [id=c2d71ff2a3ef40ba997ae05af25bfca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.364418 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:41.366956 13246 ts_tablet_manager.cc:1875] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.367460 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38453bf678748399339806d12993c88" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:41.367841 12826 tablet_service.cc:1467] Processing CreateTablet for tablet cd6245e4adce4038ad6f988877d75045 (DEFAULT_TABLE table=default.Table_2486dda60f0b4f2b9a63dbf17885bc5e [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:41.369972 12921 leader_election.cc:341] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.370146 12921 leader_election.cc:302] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.370976 13719 raft_consensus.cc:2726] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.371532 13730 raft_consensus.cc:482] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.371706 13730 raft_consensus.cc:504] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.372443 13730 leader_election.cc:288] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.373039 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 2751074a54ed4bcea6979e6ad11c2efe from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:41.373653 12814 tablet_service.cc:1467] Processing CreateTablet for tablet e63baf59dc8244daad831b2987a51cfb (DEFAULT_TABLE table=default.Table_25774424d24c4cd68a59fd89cae9b492 [id=3dcb08c9175142c7831f2224f317a453]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.374348 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b625c2bbd89746bc90b8c4c2483b3741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.374330 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9230ed2e9a49ecb0a5e5d520cd03aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.374547 12986 raft_consensus.cc:2454] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:41.375012 13057 leader_election.cc:302] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:41.378386 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:41.380278 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63baf59dc8244daad831b2987a51cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.380676 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9230ed2e9a49ecb0a5e5d520cd03aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.381064 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f28c1f55fd14aca86cb00e43b3a6b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.381323 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d09eafa82a0423a8358958e78f728e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.381553 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85365758f6a64536ab61e8349de63dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.381791 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc88443c04b42d19cc7fba339eeeaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.382382 13730 raft_consensus.cc:2781] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:41.382499 13730 raft_consensus.cc:482] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:41.382587 13730 raft_consensus.cc:3037] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:41.385459 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6245e4adce4038ad6f988877d75045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.387866 13730 raft_consensus.cc:504] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.389031 13730 leader_election.cc:288] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.389200 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9230ed2e9a49ecb0a5e5d520cd03aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:41.389496 12986 raft_consensus.cc:3037] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:41.390514 13058 leader_election.cc:341] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.391234 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 1488328703024ae8983add9e4d6afc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.392256 12986 raft_consensus.cc:2454] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:41.394637 13057 leader_election.cc:302] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:41.394951 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet a1160ab972124630a9be7abeefb4aa56 with delete_type TABLET_DATA_DELETED (replaced by tablet 44ce8749846847108edf5f554139cbd9) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.395144 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e54543d1544b1c8cff3e890e787337 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.395272 13730 raft_consensus.cc:2781] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:41.395332 13244 tablet_bootstrap.cc:492] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:41.398267 12518 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:41.399767 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9230ed2e9a49ecb0a5e5d520cd03aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:41.401424 13058 leader_election.cc:341] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.402977 13730 raft_consensus.cc:686] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:41.403340 13730 consensus_queue.cc:237] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.404434 13781 raft_consensus.cc:482] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.404917 13781 raft_consensus.cc:504] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.405941 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:41.406309 13781 leader_election.cc:288] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.406922 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d09eafa82a0423a8358958e78f728e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.408541 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.408967 12848 raft_consensus.cc:3037] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:41.409116 13058 leader_election.cc:341] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.414207 13243 ts_tablet_manager.cc:1875] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.414268 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.414273 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.417006 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.416639 13249 tablet_bootstrap.cc:492] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:41.423416 13249 ts_tablet_manager.cc:1367] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.299s user 0.031s sys 0.071s
I20240506 19:56:41.424288 13249 raft_consensus.cc:348] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.424480 13249 raft_consensus.cc:374] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.424563 13249 raft_consensus.cc:729] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.425967 13711 catalog_manager.cc:1259] Loaded cluster ID: cab80095522b47ec9cab71d12c4cac98
I20240506 19:56:41.426056 13711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: loading cluster ID for follower catalog manager: success
I20240506 19:56:41.427418 13249 consensus_queue.cc:260] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.428018 13249 ts_tablet_manager.cc:1398] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:41.428328 13249 tablet_bootstrap.cc:492] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:41.434345 12515 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:41.435480 13246 ts_tablet_manager.cc:1888] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.435624 13246 log.cc:1196] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:41.438197 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 5988b89cb52e4f858b52862138bed025 with delete_type TABLET_DATA_DELETED (replaced by tablet 4973e600038742a6bec0bb2d8f3551fe) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.438582 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7" } modify_external_catalogs: true
I20240506 19:56:41.438751 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c0ebfdd5c1cb4085babb98af4665cf5d" } modify_external_catalogs: true
I20240506 19:56:41.438858 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7" } modify_external_catalogs: true
I20240506 19:56:41.438925 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c0ebfdd5c1cb4085babb98af4665cf5d" } modify_external_catalogs: true
I20240506 19:56:41.438712 13246 ts_tablet_manager.cc:1909] T 1667a0bbd3af4d678434b068c47cc518 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.440673 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1667a0bbd3af4d678434b068c47cc518 (table default.Table_dfd40b6029db498e870d3e8a44ae6bdd [id=c4a20cff71d7496e8b07f477a0c1ea1d]) successfully deleted
I20240506 19:56:41.441249 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3ce0a12d16814dc9a767895fe39f1247" } modify_external_catalogs: true
I20240506 19:56:41.441484 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3ce0a12d16814dc9a767895fe39f1247" } modify_external_catalogs: true
I20240506 19:56:41.443636 13246 tablet_replica.cc:330] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:41.444129 13246 raft_consensus.cc:2229] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.445047 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c57d5247a9fc4628ac83a4fd916f2b53" } modify_external_catalogs: true
I20240506 19:56:41.445240 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c57d5247a9fc4628ac83a4fd916f2b53" } modify_external_catalogs: true
W20240506 19:56:41.445485 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 028055b9df1f4667bd2e54909690e327 has no consensus state
I20240506 19:56:41.446069 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6bc0ddcaf1a2494facb7e66847623940" } modify_external_catalogs: true
I20240506 19:56:41.446249 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6bc0ddcaf1a2494facb7e66847623940" } modify_external_catalogs: true
I20240506 19:56:41.447337 13246 raft_consensus.cc:2258] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.449766 12958 tablet_service.cc:1467] Processing CreateTablet for tablet bc85233b180f4b30a86bb65c86af684e (DEFAULT_TABLE table=default.Table_aabc53fdc37a4ffc89e4fe789817f8f7 [id=ca57c399a618420e8404a52e9476e8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.450269 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc85233b180f4b30a86bb65c86af684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.450415 13104 tablet_service.cc:1467] Processing CreateTablet for tablet bc85233b180f4b30a86bb65c86af684e (DEFAULT_TABLE table=default.Table_aabc53fdc37a4ffc89e4fe789817f8f7 [id=ca57c399a618420e8404a52e9476e8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.450810 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc85233b180f4b30a86bb65c86af684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.451411 13353 raft_consensus.cc:482] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.451579 13353 raft_consensus.cc:504] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.452250 13353 leader_election.cc:288] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.452826 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:41.453054 13123 raft_consensus.cc:2454] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:41.453580 12922 leader_election.cc:302] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:41.454496 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20aa06290b9a4814a006c423ccd8fc46 consensus state has no leader
I20240506 19:56:41.455017 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.455246 12986 raft_consensus.cc:2454] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:41.456070 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 64485c3c13e94055a019f2ef7be1f2fe (DEFAULT_TABLE table=default.Table_9d643b8fae2443eab31564f9f0440d60 [id=b10c43407d874387acc54a4ab3f26734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.456458 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64485c3c13e94055a019f2ef7be1f2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.457917 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 64485c3c13e94055a019f2ef7be1f2fe (DEFAULT_TABLE table=default.Table_9d643b8fae2443eab31564f9f0440d60 [id=b10c43407d874387acc54a4ab3f26734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.458287 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64485c3c13e94055a019f2ef7be1f2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.458814 12513 catalog_manager.cc:5484] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:41.459296 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_bcd38427a79c48d1badd4f2536ff238f [id=f23583e607a4419986d06a38c45bbdf5]). Expected version 2 got 0
I20240506 19:56:41.461602 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a (DEFAULT_TABLE table=default.Table_960b875a518745a38b1a1e6699ec3a2f [id=2efd55b739244f81a16c345e3d8746af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.461969 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.461939 13102 tablet_service.cc:1467] Processing CreateTablet for tablet a20016b56be94248905e0e8d6a35100f (DEFAULT_TABLE table=default.Table_dd9774c013d441f4b9fc4a0afdf7daa2 [id=28050065b4e94914afc025ddfa0bb43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.462306 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20016b56be94248905e0e8d6a35100f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.465588 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a (DEFAULT_TABLE table=default.Table_960b875a518745a38b1a1e6699ec3a2f [id=2efd55b739244f81a16c345e3d8746af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.465962 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.469043 13249 tablet_bootstrap.cc:654] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.469607 12952 tablet_service.cc:1467] Processing CreateTablet for tablet a20016b56be94248905e0e8d6a35100f (DEFAULT_TABLE table=default.Table_dd9774c013d441f4b9fc4a0afdf7daa2 [id=28050065b4e94914afc025ddfa0bb43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.469960 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20016b56be94248905e0e8d6a35100f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.471160 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 9fed2f12eda54be0879b1e9c33ca3e31 (DEFAULT_TABLE table=default.Table_7b7fe87d3acd4184b95b2dd1026980d5 [id=c074755957ef43bea0b8053ce6bffa6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.472033 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fed2f12eda54be0879b1e9c33ca3e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.472903 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_54bc24e04c80487d9a93fd01ad679d73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.473510 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54bc24e04c80487d9a93fd01ad679d73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.473832 13243 ts_tablet_manager.cc:1888] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.473793 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe8d4891c484627ac729971ff7aad3e (DEFAULT_TABLE table=default.Table_ff9dd351a2594431addcdc137481cd02 [id=92ce8151d6b5411199f196150d535634]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.473955 13243 log.cc:1196] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/0ba2365d00044c32a06f9017a207adf7
I20240506 19:56:41.474188 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe8d4891c484627ac729971ff7aad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.474475 13243 ts_tablet_manager.cc:1909] T 0ba2365d00044c32a06f9017a207adf7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:41.474853 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c0d358df6dd84d60a343a335303508eb" } new_table_name: "default.Table_bef80a208c25432493da6ad772c8bf32" modify_external_catalogs: true
I20240506 19:56:41.475337 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 311) took 3719 ms. Trace:
I20240506 19:56:41.475472 13243 rpcz_store.cc:276] 0506 19:56:37.756115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.763189 (+ 7074us) service_pool.cc:225] Handling call
0506 19:56:41.354160 (+3590971us) ts_tablet_manager.cc:1120] Deleting tablet 0ba2365d00044c32a06f9017a207adf7
0506 19:56:41.354169 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.461287 (+107118us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.473632 (+ 12345us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.475282 (+ 1650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3590683}
I20240506 19:56:41.476255 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 312) took 3707 ms. Trace:
I20240506 19:56:41.476312 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 0ba2365d00044c32a06f9017a207adf7 (table default.Table_24bc90d17d404ca3884c13c02624e651 [id=7a329c37e7684bac8bf008b1c5e53e1c]) successfully deleted
I20240506 19:56:41.476191 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f72793ccf9b146a0baad8e0836ca260c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.476361 13243 rpcz_store.cc:276] 0506 19:56:37.769276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.769383 (+ 107us) service_pool.cc:225] Handling call
0506 19:56:41.476022 (+3706639us) ts_tablet_manager.cc:1120] Deleting tablet bd2c81ac92d04ddc93cc52e5c01cbc60
0506 19:56:41.476030 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.476211 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3706258}
W20240506 19:56:41.476832 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f72793ccf9b146a0baad8e0836ca260c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.477136 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe8d4891c484627ac729971ff7aad3e (DEFAULT_TABLE table=default.Table_ff9dd351a2594431addcdc137481cd02 [id=92ce8151d6b5411199f196150d535634]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.477365 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 9fed2f12eda54be0879b1e9c33ca3e31 (DEFAULT_TABLE table=default.Table_7b7fe87d3acd4184b95b2dd1026980d5 [id=c074755957ef43bea0b8053ce6bffa6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.477712 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fed2f12eda54be0879b1e9c33ca3e31. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:41.478073 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd2c81ac92d04ddc93cc52e5c01cbc60 already in progress: creating tablet
I20240506 19:56:41.479274 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe8d4891c484627ac729971ff7aad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.479601 13245 tablet_bootstrap.cc:492] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:41.479746 13245 ts_tablet_manager.cc:1367] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.148s user 0.007s sys 0.005s
I20240506 19:56:41.481828 13353 raft_consensus.cc:2781] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:41.481930 13353 raft_consensus.cc:482] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:41.482020 13353 raft_consensus.cc:3037] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:41.480661 13245 raft_consensus.cc:348] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.483723 13245 raft_consensus.cc:374] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.483831 13245 raft_consensus.cc:729] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.481575 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 6089f4f68bfe415e8d97dfd6849b7714 (DEFAULT_TABLE table=default.Table_15b63528dc5649619d25b71595773037 [id=30577c164a974b99a479276640cca87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.484071 13245 consensus_queue.cc:260] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.484257 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6089f4f68bfe415e8d97dfd6849b7714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.481818 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 76e6f7555dfa4122ac00bae7b9e733fe (DEFAULT_TABLE table=default.Table_dbebd827d8ed405bae594e5adc6c14a9 [id=9006912cbc764b9a8f90a11d963cf02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.486882 13246 ts_tablet_manager.cc:1875] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.490098 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 6089f4f68bfe415e8d97dfd6849b7714 (DEFAULT_TABLE table=default.Table_15b63528dc5649619d25b71595773037 [id=30577c164a974b99a479276640cca87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.490774 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6089f4f68bfe415e8d97dfd6849b7714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.495790 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 76e6f7555dfa4122ac00bae7b9e733fe (DEFAULT_TABLE table=default.Table_dbebd827d8ed405bae594e5adc6c14a9 [id=9006912cbc764b9a8f90a11d963cf02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.496346 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e6f7555dfa4122ac00bae7b9e733fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.498715 12960 tablet_service.cc:1467] Processing CreateTablet for tablet df04fb90230d4bf4a060db993adafd14 (DEFAULT_TABLE table=default.Table_cf03515d64254430b39e443cedd953f9 [id=1b9e541b5a194bc6a8ed9e1f61fbc5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.499114 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df04fb90230d4bf4a060db993adafd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.499850 12962 tablet_service.cc:1467] Processing CreateTablet for tablet df160fd5d8ce4bc192913ce702f8f478 (DEFAULT_TABLE table=default.Table_b9c18afe9e154517909885ea89fefcc2 [id=1f7e95047b4840499fa89ae4b887e279]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.500202 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df160fd5d8ce4bc192913ce702f8f478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.502555 12963 tablet_service.cc:1467] Processing CreateTablet for tablet de26793c006440dc9aecc07963ddb85e (DEFAULT_TABLE table=default.Table_dfcabd286a794ef19121c86363912db0 [id=c4484634c74946b39ee80b8ad1495c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.503345 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de26793c006440dc9aecc07963ddb85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.504145 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_04311b0062d745e3ae935a1abe99343e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.504741 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04311b0062d745e3ae935a1abe99343e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.495764 13353 raft_consensus.cc:504] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.505430 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 11b00381c6aa42c9b969410b51ebe81e (DEFAULT_TABLE table=default.Table_da2ec44bc10845a7a4fbf93f4916c652 [id=cf1b4de7eeaa4aabaa3dc7c5160ec86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.507439 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2417a535329341d8bb9f8218770f890d
I20240506 19:56:41.508343 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ac80e9e0789c4cbaab8826eb3c0bc621" } new_table_name: "default.Table_7ec0af39e86c4ffeb0622785e6a817be" modify_external_catalogs: true
I20240506 19:56:41.508649 13244 tablet_bootstrap.cc:654] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.509514 12822 tablet_service.cc:1467] Processing CreateTablet for tablet bc85233b180f4b30a86bb65c86af684e (DEFAULT_TABLE table=default.Table_aabc53fdc37a4ffc89e4fe789817f8f7 [id=ca57c399a618420e8404a52e9476e8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.510833 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.515036 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ff3049a9fa8049e5b84230e5e6e8245c
I20240506 19:56:41.516337 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b6fa6add23b8403ba5a0a65ac6752a5b
W20240506 19:56:41.519573 12921 leader_election.cc:341] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.520027 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 458314e4653748e48cf646c72373fc3e
I20240506 19:56:41.521673 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 986f9c83b04f41b49e9b78fce731a90f (DEFAULT_TABLE table=default.Table_5a18ae8689874543a57ab917b1efaff0 [id=577dd46fb1b44fd380574a6cd0ba2695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.522310 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 2417a535329341d8bb9f8218770f890d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:41.530579 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0aa8f9aaca444c839a88c07f58cb6ad6 consensus state has no leader
I20240506 19:56:41.530719 13353 leader_election.cc:288] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.530814 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc85233b180f4b30a86bb65c86af684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.530987 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ee59a76d52fc48d7bcf3256c359d3c02
I20240506 19:56:41.531320 12966 tablet_service.cc:1467] Processing CreateTablet for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 (DEFAULT_TABLE table=default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.534598 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 64485c3c13e94055a019f2ef7be1f2fe (DEFAULT_TABLE table=default.Table_9d643b8fae2443eab31564f9f0440d60 [id=b10c43407d874387acc54a4ab3f26734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.535029 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64485c3c13e94055a019f2ef7be1f2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.537742 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:41.542235 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a (DEFAULT_TABLE table=default.Table_960b875a518745a38b1a1e6699ec3a2f [id=2efd55b739244f81a16c345e3d8746af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.543030 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 9fed2f12eda54be0879b1e9c33ca3e31 (DEFAULT_TABLE table=default.Table_7b7fe87d3acd4184b95b2dd1026980d5 [id=c074755957ef43bea0b8053ce6bffa6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.543416 12825 tablet_service.cc:1467] Processing CreateTablet for tablet a20016b56be94248905e0e8d6a35100f (DEFAULT_TABLE table=default.Table_dd9774c013d441f4b9fc4a0afdf7daa2 [id=28050065b4e94914afc025ddfa0bb43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.547135 13711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: acquiring CA information for follower catalog manager: success
I20240506 19:56:41.547827 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.548118 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fed2f12eda54be0879b1e9c33ca3e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.549562 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20016b56be94248905e0e8d6a35100f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.555629 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe8d4891c484627ac729971ff7aad3e (DEFAULT_TABLE table=default.Table_ff9dd351a2594431addcdc137481cd02 [id=92ce8151d6b5411199f196150d535634]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.560971 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe8d4891c484627ac729971ff7aad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.563367 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cef67076343c4a73b29b2d8091257dd2" } new_table_name: "default.Table_d12f952acabc497c9f8f644888a0a361" modify_external_catalogs: true
I20240506 19:56:41.563310 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0ea05891ea5e48a5b2a179a245749dc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.563948 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ea05891ea5e48a5b2a179a245749dc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:41.565339 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4973e600038742a6bec0bb2d8f3551fe consensus state has no leader
I20240506 19:56:41.570272 13057 leader_election.cc:302] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.571341 12964 tablet_service.cc:1467] Processing CreateTablet for tablet b6fa6add23b8403ba5a0a65ac6752a5b (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.571388 12810 tablet_service.cc:1467] Processing CreateTablet for tablet de26793c006440dc9aecc07963ddb85e (DEFAULT_TABLE table=default.Table_dfcabd286a794ef19121c86363912db0 [id=c4484634c74946b39ee80b8ad1495c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.571753 13353 raft_consensus.cc:482] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.577909 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 458314e4653748e48cf646c72373fc3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.577960 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet b6fa6add23b8403ba5a0a65ac6752a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.578394 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet ee59a76d52fc48d7bcf3256c359d3c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:41.578639 12921 leader_election.cc:341] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.579053 13629 consensus_queue.cc:1035] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:41.581032 13353 raft_consensus.cc:504] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:41.581406 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 68363.5us
I20240506 19:56:41.581744 13353 leader_election.cc:288] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.581439 13245 ts_tablet_manager.cc:1398] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.091s user 0.001s sys 0.001s
I20240506 19:56:41.583662 13244 tablet_bootstrap.cc:492] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:41.583858 13244 ts_tablet_manager.cc:1367] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.212s user 0.005s sys 0.000s
I20240506 19:56:41.580464 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3049a9fa8049e5b84230e5e6e8245c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.585705 13783 raft_consensus.cc:482] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.585927 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 6089f4f68bfe415e8d97dfd6849b7714 (DEFAULT_TABLE table=default.Table_15b63528dc5649619d25b71595773037 [id=30577c164a974b99a479276640cca87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.585940 13783 raft_consensus.cc:504] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.586802 13783 leader_election.cc:288] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.586962 13249 tablet_bootstrap.cc:492] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:41.587544 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 310) took 3822 ms. Trace:
I20240506 19:56:41.588626 12818 tablet_service.cc:1467] Processing CreateTablet for tablet df04fb90230d4bf4a060db993adafd14 (DEFAULT_TABLE table=default.Table_cf03515d64254430b39e443cedd953f9 [id=1b9e541b5a194bc6a8ed9e1f61fbc5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.590456 13249 ts_tablet_manager.cc:1367] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.159s user 0.003s sys 0.000s
I20240506 19:56:41.593158 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b00381c6aa42c9b969410b51ebe81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.594532 13730 raft_consensus.cc:2781] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:41.595191 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e6f7555dfa4122ac00bae7b9e733fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.599815 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 76e6f7555dfa4122ac00bae7b9e733fe (DEFAULT_TABLE table=default.Table_dbebd827d8ed405bae594e5adc6c14a9 [id=9006912cbc764b9a8f90a11d963cf02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.601982 13245 tablet_bootstrap.cc:492] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:41.602564 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de26793c006440dc9aecc07963ddb85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.602521 13730 raft_consensus.cc:482] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:41.602653 13730 raft_consensus.cc:3037] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:41.602984 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 11b00381c6aa42c9b969410b51ebe81e (DEFAULT_TABLE table=default.Table_da2ec44bc10845a7a4fbf93f4916c652 [id=cf1b4de7eeaa4aabaa3dc7c5160ec86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.603665 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 986f9c83b04f41b49e9b78fce731a90f (DEFAULT_TABLE table=default.Table_5a18ae8689874543a57ab917b1efaff0 [id=577dd46fb1b44fd380574a6cd0ba2695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.603726 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66f1e46cce47639fe644fb4ed2e5c0" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.603675 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:41.604391 12921 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.594310 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 11b00381c6aa42c9b969410b51ebe81e (DEFAULT_TABLE table=default.Table_da2ec44bc10845a7a4fbf93f4916c652 [id=cf1b4de7eeaa4aabaa3dc7c5160ec86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.606065 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:41.608165 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 458314e4653748e48cf646c72373fc3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.608440 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3049a9fa8049e5b84230e5e6e8245c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.608788 12519 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:41.608958 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet ee59a76d52fc48d7bcf3256c359d3c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.609021 13104 tablet_service.cc:1467] Processing CreateTablet for tablet df04fb90230d4bf4a060db993adafd14 (DEFAULT_TABLE table=default.Table_cf03515d64254430b39e443cedd953f9 [id=1b9e541b5a194bc6a8ed9e1f61fbc5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.609515 12828 tablet_service.cc:1467] Processing CreateTablet for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 (DEFAULT_TABLE table=default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.609863 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet b6fa6add23b8403ba5a0a65ac6752a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.609956 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b00381c6aa42c9b969410b51ebe81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.610617 12815 tablet_service.cc:1467] Processing CreateTablet for tablet b6fa6add23b8403ba5a0a65ac6752a5b (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.611246 12823 tablet_service.cc:1467] Processing CreateTablet for tablet df160fd5d8ce4bc192913ce702f8f478 (DEFAULT_TABLE table=default.Table_b9c18afe9e154517909885ea89fefcc2 [id=1f7e95047b4840499fa89ae4b887e279]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.611882 13095 tablet_service.cc:1467] Processing CreateTablet for tablet df160fd5d8ce4bc192913ce702f8f478 (DEFAULT_TABLE table=default.Table_b9c18afe9e154517909885ea89fefcc2 [id=1f7e95047b4840499fa89ae4b887e279]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:41.613862 13058 leader_election.cc:341] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.616794 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df04fb90230d4bf4a060db993adafd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.617092 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6fa6add23b8403ba5a0a65ac6752a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.617374 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df160fd5d8ce4bc192913ce702f8f478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.617609 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986f9c83b04f41b49e9b78fce731a90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.617882 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ece0bd3a3b47ef88a0b4910f0efae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.618124 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6089f4f68bfe415e8d97dfd6849b7714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.618333 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b00381c6aa42c9b969410b51ebe81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.618728 13087 tablet_service.cc:1467] Processing CreateTablet for tablet de26793c006440dc9aecc07963ddb85e (DEFAULT_TABLE table=default.Table_dfcabd286a794ef19121c86363912db0 [id=c4484634c74946b39ee80b8ad1495c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.621567 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 2417a535329341d8bb9f8218770f890d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.621698 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df04fb90230d4bf4a060db993adafd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.621930 13244 raft_consensus.cc:348] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.622299 13096 tablet_service.cc:1467] Processing CreateTablet for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 (DEFAULT_TABLE table=default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.622195 13244 raft_consensus.cc:374] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.622534 13244 raft_consensus.cc:729] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.622844 13244 consensus_queue.cc:260] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.623550 13244 ts_tablet_manager.cc:1398] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.040s user 0.002s sys 0.001s
I20240506 19:56:41.623950 13244 tablet_bootstrap.cc:492] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:41.624771 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 986f9c83b04f41b49e9b78fce731a90f (DEFAULT_TABLE table=default.Table_5a18ae8689874543a57ab917b1efaff0 [id=577dd46fb1b44fd380574a6cd0ba2695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.621094 13093 tablet_service.cc:1467] Processing CreateTablet for tablet b6fa6add23b8403ba5a0a65ac6752a5b (DEFAULT_TABLE table=default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.628309 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de26793c006440dc9aecc07963ddb85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.628618 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6fa6add23b8403ba5a0a65ac6752a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.629417 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df160fd5d8ce4bc192913ce702f8f478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.630671 13242 ts_tablet_manager.cc:1888] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.630856 13242 log.cc:1196] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a16704567ef6454eaef886fb9cd599ce
I20240506 19:56:41.631315 13242 ts_tablet_manager.cc:1909] T a16704567ef6454eaef886fb9cd599ce P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:41.631701 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 293) took 3966 ms. Trace:
I20240506 19:56:41.631830 13242 rpcz_store.cc:276] 0506 19:56:37.664765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.664805 (+ 40us) service_pool.cc:225] Handling call
0506 19:56:41.050716 (+3385911us) ts_tablet_manager.cc:1120] Deleting tablet a16704567ef6454eaef886fb9cd599ce
0506 19:56:41.074980 (+ 24264us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.510359 (+435379us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.628953 (+118594us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.631638 (+ 2685us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":723,"tablet-delete.queue_time_us":3385729}
I20240506 19:56:41.632592 13242 tablet_replica.cc:330] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:41.632830 13242 raft_consensus.cc:2229] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.633035 13242 raft_consensus.cc:2258] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.634119 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet 2417a535329341d8bb9f8218770f890d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.634356 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3049a9fa8049e5b84230e5e6e8245c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.635269 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ece0bd3a3b47ef88a0b4910f0efae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.635566 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986f9c83b04f41b49e9b78fce731a90f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:41.637385 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45b2a4720f6a46e4b0412a58c1e26bf7 consensus state has no leader
I20240506 19:56:41.638814 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a16704567ef6454eaef886fb9cd599ce (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:41.642318 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d01da29ce66546ab88b2eba38a69c4d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.643023 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d01da29ce66546ab88b2eba38a69c4d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.643131 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet ee59a76d52fc48d7bcf3256c359d3c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.644662 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet b6fa6add23b8403ba5a0a65ac6752a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.647053 13245 tablet_bootstrap.cc:654] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.647385 13730 raft_consensus.cc:504] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.649514 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5f73ddfe064b4a9ea98eaa48ce44334f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.650070 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f73ddfe064b4a9ea98eaa48ce44334f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.650168 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.650480 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 458314e4653748e48cf646c72373fc3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.651826 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1d3c02a3c0e64178939ba1d042908501"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.652364 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d3c02a3c0e64178939ba1d042908501 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.652307 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4180588e9c5d43cd95909d0f1cb7a140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.652805 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4180588e9c5d43cd95909d0f1cb7a140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.653506 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9d643b8fae2443eab31564f9f0440d60" } new_table_name: "default.Table_f6ec20e42cd94faebb5b13d490069e9d" modify_external_catalogs: true
I20240506 19:56:41.654167 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_071b75a659614286996e871c603f9dcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.654582 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_071b75a659614286996e871c603f9dcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.656313 13730 leader_election.cc:288] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:41.659300 12922 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:41.659621 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.659940 13249 raft_consensus.cc:348] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.660727 13123 raft_consensus.cc:2379] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:41.660717 13243 rpcz_store.cc:276] 0506 19:56:37.755973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.777360 (+ 21387us) service_pool.cc:225] Handling call
0506 19:56:41.533342 (+3755982us) ts_tablet_manager.cc:1120] Deleting tablet 0114edac976844619e438c17bb959010
0506 19:56:41.533351 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.578284 (+ 44933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3648318}
W20240506 19:56:41.661237 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0114edac976844619e438c17bb959010 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0114edac976844619e438c17bb959010 already in progress: creating tablet
I20240506 19:56:41.662261 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986f9c83b04f41b49e9b78fce731a90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.665380 12922 leader_election.cc:302] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.666672 12922 leader_election.cc:302] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.667114 13353 raft_consensus.cc:2726] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.667718 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:41.668107 13629 raft_consensus.cc:2726] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.668370 12988 raft_consensus.cc:2379] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a9c37db9d4f41cd93820a1c81f48d4d in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:56:41.668769 12986 raft_consensus.cc:1261] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Refusing update from remote peer 8a9c37db9d4f41cd93820a1c81f48d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:41.669088 13249 raft_consensus.cc:374] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.669183 13249 raft_consensus.cc:729] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.669799 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ece0bd3a3b47ef88a0b4910f0efae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.669772 13249 consensus_queue.cc:260] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.670362 13249 ts_tablet_manager.cc:1398] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
W20240506 19:56:41.671367 13057 leader_election.cc:341] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.673655 13249 tablet_bootstrap.cc:492] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:41.673918 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6fa6add23b8403ba5a0a65ac6752a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.680094 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_db02a40f1e43425ea3ed5c28eaba58b9" } new_table_name: "default.Table_f48001b944b84606b9e9e95a00e8a6a7" modify_external_catalogs: true
I20240506 19:56:41.681823 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_857904c3c3954ef5a113de49478bd27f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.682358 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857904c3c3954ef5a113de49478bd27f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.696588 13730 raft_consensus.cc:482] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.696789 13730 raft_consensus.cc:504] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.697960 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 318) took 3526 ms. Trace:
I20240506 19:56:41.692696 13102 tablet_service.cc:1467] Processing CreateTablet for tablet cade6b1c5a034b23aa33b2349e1dab95 (DEFAULT_TABLE table=default.Table_8507072bb8ea467982eab99c978a7ac8 [id=3bcbaf5c0f19445789fc1a1b3dacd629]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.699232 13057 leader_election.cc:302] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.700093 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
W20240506 19:56:41.701049 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64485c3c13e94055a019f2ef7be1f2fe consensus state has no leader
W20240506 19:56:41.710318 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b4ad903e75547fa829b82cc722fef1e consensus state has no leader
I20240506 19:56:41.711133 13244 tablet_bootstrap.cc:654] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.712882 13243 rpcz_store.cc:276] 0506 19:56:38.135220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.135276 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:41.661759 (+3526483us) ts_tablet_manager.cc:1120] Deleting tablet 00de09ddcdd948a6a3500a6c9d814c14
0506 19:56:41.661860 (+ 101us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.661961 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3526184}
I20240506 19:56:41.713220 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfcabd286a794ef19121c86363912db0" } modify_external_catalogs: true
I20240506 19:56:41.713454 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfcabd286a794ef19121c86363912db0" } modify_external_catalogs: true
I20240506 19:56:41.713739 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1488328703024ae8983add9e4d6afc8f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.713862 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 320) took 3578 ms. Trace:
W20240506 19:56:41.714107 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 00de09ddcdd948a6a3500a6c9d814c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00de09ddcdd948a6a3500a6c9d814c14 already in progress: creating tablet
I20240506 19:56:41.714587 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet 9e3d13322b4a4a2db3b6d425679f68ed from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:41.715286 13243 rpcz_store.cc:276] 0506 19:56:38.135482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.135636 (+ 154us) service_pool.cc:225] Handling call
0506 19:56:41.713693 (+3578057us) ts_tablet_manager.cc:1120] Deleting tablet 6b59ad067c274fddbba5f71e89480dec
0506 19:56:41.713703 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.713819 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576566}
W20240506 19:56:41.715860 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 6b59ad067c274fddbba5f71e89480dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b59ad067c274fddbba5f71e89480dec already in progress: creating tablet
I20240506 19:56:41.716351 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cade6b1c5a034b23aa33b2349e1dab95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.719843 13783 raft_consensus.cc:2726] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.719935 13730 leader_election.cc:288] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.724789 13783 consensus_queue.cc:1035] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20240506 19:56:41.725955 13057 leader_election.cc:341] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.731034 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet de26793c006440dc9aecc07963ddb85e
I20240506 19:56:41.732318 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:41.732842 12519 catalog_manager.cc:6554] ReplaceTablet: tablet 9e3d13322b4a4a2db3b6d425679f68ed deleted and replaced by tablet a03f574aec984b83ab596e92a6f323fc
I20240506 19:56:41.735074 13249 tablet_bootstrap.cc:654] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:41.735647 12922 consensus_peers.cc:485] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6156c6f582b747de973b357b6d071a46. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:41.736625 13246 ts_tablet_manager.cc:1888] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.737176 13246 log.cc:1196] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/4f91888e88234cbdbd00f48dc1574653
I20240506 19:56:41.738257 13242 ts_tablet_manager.cc:1875] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.738468 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e6f7555dfa4122ac00bae7b9e733fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.737587 12818 tablet_service.cc:1467] Processing CreateTablet for tablet cade6b1c5a034b23aa33b2349e1dab95 (DEFAULT_TABLE table=default.Table_8507072bb8ea467982eab99c978a7ac8 [id=3bcbaf5c0f19445789fc1a1b3dacd629]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.746115 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1488328703024ae8983add9e4d6afc8f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.746457 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed with delete_type TABLET_DATA_DELETED (replaced by tablet a03f574aec984b83ab596e92a6f323fc) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.749560 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 319) took 3601 ms. Trace:
I20240506 19:56:41.751279 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet de26793c006440dc9aecc07963ddb85e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.751777 12962 tablet_service.cc:1467] Processing CreateTablet for tablet cade6b1c5a034b23aa33b2349e1dab95 (DEFAULT_TABLE table=default.Table_8507072bb8ea467982eab99c978a7ac8 [id=3bcbaf5c0f19445789fc1a1b3dacd629]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.752709 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed with delete_type TABLET_DATA_DELETED (replaced by tablet a03f574aec984b83ab596e92a6f323fc) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.753237 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet de26793c006440dc9aecc07963ddb85e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:41.753484 13058 consensus_peers.cc:485] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:41.753747 13058 leader_election.cc:341] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.753990 13246 ts_tablet_manager.cc:1909] T 4f91888e88234cbdbd00f48dc1574653 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.754343 13058 leader_election.cc:302] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.754356 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet de26793c006440dc9aecc07963ddb85e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.754786 13783 raft_consensus.cc:2726] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.755436 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cade6b1c5a034b23aa33b2349e1dab95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.769829 13246 tablet_replica.cc:330] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:41.770087 13246 raft_consensus.cc:2229] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.770500 13246 raft_consensus.cc:2258] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:41.770740 12922 consensus_peers.cc:485] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40ece0ad4df247eb9d380ed1c755ac99. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:41.771085 13246 ts_tablet_manager.cc:1875] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.777411 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cade6b1c5a034b23aa33b2349e1dab95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.777383 13243 rpcz_store.cc:276] 0506 19:56:38.135345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.135611 (+ 266us) service_pool.cc:225] Handling call
0506 19:56:41.717361 (+3581750us) ts_tablet_manager.cc:1120] Deleting tablet 1372be71e1d141ed89e58cdb806431a4
0506 19:56:41.721378 (+ 4017us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.736442 (+ 15064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3478026}
W20240506 19:56:41.778014 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1372be71e1d141ed89e58cdb806431a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1372be71e1d141ed89e58cdb806431a4 already in progress: creating tablet
I20240506 19:56:41.778126 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 334) took 3526 ms. Trace:
I20240506 19:56:41.778275 13243 rpcz_store.cc:276] 0506 19:56:38.251371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.251653 (+ 282us) service_pool.cc:225] Handling call
0506 19:56:41.777941 (+3526288us) ts_tablet_manager.cc:1120] Deleting tablet f3e7329b22414da886dee4e1ec6bf3ad
0506 19:56:41.777958 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.778081 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3290822}
W20240506 19:56:41.778677 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f3e7329b22414da886dee4e1ec6bf3ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3e7329b22414da886dee4e1ec6bf3ad already in progress: creating tablet
I20240506 19:56:41.778779 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 335) took 3343 ms. Trace:
I20240506 19:56:41.778900 13243 rpcz_store.cc:276] 0506 19:56:38.435229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.436774 (+ 1545us) service_pool.cc:225] Handling call
0506 19:56:41.778646 (+3341872us) ts_tablet_manager.cc:1120] Deleting tablet ceeae70e0d1144ea9e3e0f8727c15e6b
0506 19:56:41.778655 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.778739 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3290436}
W20240506 19:56:41.779240 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ceeae70e0d1144ea9e3e0f8727c15e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceeae70e0d1144ea9e3e0f8727c15e6b already in progress: creating tablet
I20240506 19:56:41.779400 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 349) took 2956 ms. Trace:
I20240506 19:56:41.779511 13243 rpcz_store.cc:276] 0506 19:56:38.823057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.823379 (+ 322us) service_pool.cc:225] Handling call
0506 19:56:41.779228 (+2955849us) ts_tablet_manager.cc:1120] Deleting tablet 607b3fc2c17b4793af9b84c19066ec5e
0506 19:56:41.779244 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.779360 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954967}
W20240506 19:56:41.779845 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 607b3fc2c17b4793af9b84c19066ec5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607b3fc2c17b4793af9b84c19066ec5e already in progress: creating tablet
I20240506 19:56:41.779942 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 347) took 2957 ms. Trace:
I20240506 19:56:41.780046 13243 rpcz_store.cc:276] 0506 19:56:38.822761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.825718 (+ 2957us) service_pool.cc:225] Handling call
0506 19:56:41.779817 (+2954099us) ts_tablet_manager.cc:1120] Deleting tablet 2b5a46b91cb742a994496648fbf87aa4
0506 19:56:41.779827 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.779907 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2953768}
W20240506 19:56:41.780388 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2b5a46b91cb742a994496648fbf87aa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b5a46b91cb742a994496648fbf87aa4 already in progress: creating tablet
I20240506 19:56:41.780489 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 346) took 2957 ms. Trace:
I20240506 19:56:41.780591 13243 rpcz_store.cc:276] 0506 19:56:38.822612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.826112 (+ 3500us) service_pool.cc:225] Handling call
0506 19:56:41.780376 (+2954264us) ts_tablet_manager.cc:1120] Deleting tablet 17da01620b8b41008421b977455e0c9a
0506 19:56:41.780385 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.780455 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2953955}
W20240506 19:56:41.780910 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 17da01620b8b41008421b977455e0c9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17da01620b8b41008421b977455e0c9a already in progress: creating tablet
I20240506 19:56:41.781064 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 348) took 2958 ms. Trace:
I20240506 19:56:41.781164 13243 rpcz_store.cc:276] 0506 19:56:38.822920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.826472 (+ 3552us) service_pool.cc:225] Handling call
0506 19:56:41.780892 (+2954420us) ts_tablet_manager.cc:1120] Deleting tablet 971db6ac24d54efca95695ed1c90d4ed
0506 19:56:41.780902 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.781026 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954162}
I20240506 19:56:41.781579 13243 tablet_replica.cc:330] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.781831 13243 raft_consensus.cc:2229] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:41.781960 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 971db6ac24d54efca95695ed1c90d4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 971db6ac24d54efca95695ed1c90d4ed already in progress: creating tablet
I20240506 19:56:41.782063 13243 raft_consensus.cc:2258] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.791908 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
I20240506 19:56:41.793552 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 4f91888e88234cbdbd00f48dc1574653 (table default.Table_4b6f407e4a464024aed5a1fb84c49849 [id=c7b8c2f7b9f74051b8207e0e127d930d]) successfully deleted
I20240506 19:56:41.793454 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2751074a54ed4bcea6979e6ad11c2efe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:41.795183 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_11b5bac0d01243f4a527f32ba62d5a38" } modify_external_catalogs: true
I20240506 19:56:41.795446 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_11b5bac0d01243f4a527f32ba62d5a38" } modify_external_catalogs: true
I20240506 19:56:41.795497 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c9a95dafc84a43e5926a1f04aa925c19" } new_table_name: "default.Table_9898f6715c8a4ed183046f6a9e930d11" modify_external_catalogs: true
I20240506 19:56:41.795655 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_604f6c5cb5214c97b981184d3202d345" } modify_external_catalogs: true
I20240506 19:56:41.795794 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_604f6c5cb5214c97b981184d3202d345" } modify_external_catalogs: true
I20240506 19:56:41.798406 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4674aa2f59294605bc41516bbae12158"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.798964 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4674aa2f59294605bc41516bbae12158 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.801793 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ca9a4ec248934294830f719bc5d46204"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.802451 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca9a4ec248934294830f719bc5d46204 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:41.805130 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bff7050bbc2e47cf823d25ec10b89718 consensus state has no leader
I20240506 19:56:41.805548 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f3d57249f30a44dab3384092ec20b515
I20240506 19:56:41.808516 13249 tablet_bootstrap.cc:492] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:41.808648 13249 ts_tablet_manager.cc:1367] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.138s user 0.007s sys 0.004s
I20240506 19:56:41.809608 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d57249f30a44dab3384092ec20b515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.809543 13249 raft_consensus.cc:348] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.809823 13249 raft_consensus.cc:374] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.810138 12966 tablet_service.cc:1467] Processing CreateTablet for tablet a568db06324a4fd1b58a29ef0831eb71 (DEFAULT_TABLE table=default.Table_46aaa890b4ce4d02919bab7b358c7a79 [id=5bf872e62221484ca77c5d51ae682edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.810268 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 55d9f32618b64ffa92b5a868081c36a6 (DEFAULT_TABLE table=default.Table_df0ba23556a64063af810291bf81830e [id=b0c9bae470ea4be1a475a6d536632412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.810348 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 5b375490272a4dec9a743f6458f6d6de (DEFAULT_TABLE table=default.Table_90d5ee072fbd4121ae52144e32010742 [id=c749aef4b9e0450fada39042d70c1917]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.809942 13249 raft_consensus.cc:729] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.810750 13249 consensus_queue.cc:260] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.810881 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 028055b9df1f4667bd2e54909690e327 (DEFAULT_TABLE table=default.Table_51c8c97056294104afce55e84e26a550 [id=4a72c498da064f31a0c5437f8e0d1048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.811812 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4ad903e75547fa829b82cc722fef1e (DEFAULT_TABLE table=default.Table_f48001b944b84606b9e9e95a00e8a6a7 [id=e24a1c603b7e4f6aabcbeb5dd2df1f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.811878 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 744864043f9347e0a13f8c2a7d6b147f (DEFAULT_TABLE table=default.Table_f7049cb5beca4e30b1f0a315789a1e1e [id=079ee8d63b57495c8432a5dc9e100947]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.812294 12810 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed with delete_type TABLET_DATA_DELETED (replaced by tablet a03f574aec984b83ab596e92a6f323fc) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.813546 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_aabc53fdc37a4ffc89e4fe789817f8f7" } modify_external_catalogs: true
I20240506 19:56:41.813764 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_aabc53fdc37a4ffc89e4fe789817f8f7" } modify_external_catalogs: true
I20240506 19:56:41.814090 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_978baf79e99f4eb8840e5030d3f7b93c" } modify_external_catalogs: true
I20240506 19:56:41.814054 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_730c6ad3babb49aaba5d81f48381daec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.814267 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_978baf79e99f4eb8840e5030d3f7b93c" } modify_external_catalogs: true
W20240506 19:56:41.814615 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_730c6ad3babb49aaba5d81f48381daec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.815105 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_58c38385950d438c9f8fd0451fdc858d" } new_table_name: "default.Table_bf189fc046d248199cdc55fe6c701660" modify_external_catalogs: true
I20240506 19:56:41.815793 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a568db06324a4fd1b58a29ef0831eb71. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:41.816642 13058 leader_election.cc:341] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.817508 13058 leader_election.cc:302] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.817590 13716 raft_consensus.cc:482] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.817734 13716 raft_consensus.cc:504] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.817922 13730 raft_consensus.cc:2726] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.818455 13716 leader_election.cc:288] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.818517 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 55d9f32618b64ffa92b5a868081c36a6 (DEFAULT_TABLE table=default.Table_df0ba23556a64063af810291bf81830e [id=b0c9bae470ea4be1a475a6d536632412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.819177 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393073bca2a04d15b3e583700262f3e0" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.819543 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d9f32618b64ffa92b5a868081c36a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:41.819721 12781 leader_election.cc:341] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.822700 13629 raft_consensus.cc:482] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.822883 13629 raft_consensus.cc:504] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.823462 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d9f32618b64ffa92b5a868081c36a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.823578 13629 leader_election.cc:288] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.824277 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5988b89cb52e4f858b52862138bed025" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:41.824286 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5988b89cb52e4f858b52862138bed025" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:41.824867 12921 leader_election.cc:341] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:41.826925 12922 leader_election.cc:341] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.827168 12922 leader_election.cc:302] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.827739 13629 raft_consensus.cc:2726] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.830302 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 2b949c48806d45c3a74efbc8503c0bdb (DEFAULT_TABLE table=default.Table_9bca56a380f341c28cf5c3f683024098 [id=798e81e44a5a46ec962f48302ab404a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.830765 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b949c48806d45c3a74efbc8503c0bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.831238 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028055b9df1f4667bd2e54909690e327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.833891 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4ad903e75547fa829b82cc722fef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.811159 13243 ts_tablet_manager.cc:1875] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.837708 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744864043f9347e0a13f8c2a7d6b147f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.803732 13793 tablet.cc:1722] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:41.841872 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 44ce8749846847108edf5f554139cbd9 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.842290 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ce8749846847108edf5f554139cbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.842235 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb9c05d703f417780e77359e19e542e (DEFAULT_TABLE table=default.Table_c4128d96524a4c20b2f9a7ceedccfe96 [id=e6b450dbd2dd45308a4309aea08be26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.842582 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb9c05d703f417780e77359e19e542e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.842813 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4 (DEFAULT_TABLE table=default.Table_57c82ed0d1044d20b72a0e9341b299c5 [id=ea7a610430e34ca2b3ab4868800722d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.843144 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.844931 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bc85233b180f4b30a86bb65c86af684e
W20240506 19:56:41.846837 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 796628a0859c4511a8cb35e495dc8886 consensus state has no leader
I20240506 19:56:41.848356 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f3a8531a1793404699532295164a3895
W20240506 19:56:41.848716 12922 consensus_peers.cc:485] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dc0ac37f67942fc8a31d8d43c839246. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:41.849305 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet bc85233b180f4b30a86bb65c86af684e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.856848 13249 ts_tablet_manager.cc:1398] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20240506 19:56:41.864599 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b375490272a4dec9a743f6458f6d6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.868574 13249 tablet_bootstrap.cc:492] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:41.870965 13249 tablet_bootstrap.cc:654] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.873024 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a8531a1793404699532295164a3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.873065 13711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 19:56:41.879087 13243 ts_tablet_manager.cc:1888] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.881458 13243 log.cc:1196] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/393073bca2a04d15b3e583700262f3e0
I20240506 19:56:41.884047 13243 ts_tablet_manager.cc:1909] T 393073bca2a04d15b3e583700262f3e0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:41.884094 12921 consensus_peers.cc:485] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:41.885443 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 350) took 3052 ms. Trace:
I20240506 19:56:41.885779 13629 raft_consensus.cc:482] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.886551 13249 tablet_bootstrap.cc:492] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:41.886855 13629 raft_consensus.cc:504] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.887109 13249 ts_tablet_manager.cc:1367] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240506 19:56:41.886824 13243 rpcz_store.cc:276] 0506 19:56:38.832524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.835328 (+ 2804us) service_pool.cc:225] Handling call
0506 19:56:41.781509 (+2946181us) ts_tablet_manager.cc:1120] Deleting tablet 393073bca2a04d15b3e583700262f3e0
0506 19:56:41.781519 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.860118 (+ 78599us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.878977 (+ 18859us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.885381 (+ 6404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2935598}
I20240506 19:56:41.887694 13629 leader_election.cc:288] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.887979 13249 raft_consensus.cc:348] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.888195 13249 raft_consensus.cc:374] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.888290 13249 raft_consensus.cc:729] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.875187 13242 ts_tablet_manager.cc:1888] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:41.891688 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 393073bca2a04d15b3e583700262f3e0 (table default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c [id=fd3051399ebf4aa2b31bbd709b120d48]) successfully deleted
I20240506 19:56:41.892403 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d57249f30a44dab3384092ec20b515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:41.892643 13242 log.cc:1196] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a80834187f874661b611b43065eee943
I20240506 19:56:41.893209 13242 ts_tablet_manager.cc:1909] T a80834187f874661b611b43065eee943 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:41.893669 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 294) took 4228 ms. Trace:
I20240506 19:56:41.893582 13104 tablet_service.cc:1467] Processing CreateTablet for tablet a568db06324a4fd1b58a29ef0831eb71 (DEFAULT_TABLE table=default.Table_46aaa890b4ce4d02919bab7b358c7a79 [id=5bf872e62221484ca77c5d51ae682edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.893802 13242 rpcz_store.cc:276] 0506 19:56:37.665214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.665265 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:41.632137 (+3966872us) ts_tablet_manager.cc:1120] Deleting tablet a80834187f874661b611b43065eee943
0506 19:56:41.632146 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.820203 (+188057us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.875047 (+ 54844us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.893598 (+ 18551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3966644}
I20240506 19:56:41.894347 13730 raft_consensus.cc:482] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.894407 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 300) took 4208 ms. Trace:
I20240506 19:56:41.894527 13242 rpcz_store.cc:276] 0506 19:56:37.685945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.686024 (+ 79us) service_pool.cc:225] Handling call
0506 19:56:41.894167 (+4208143us) ts_tablet_manager.cc:1120] Deleting tablet 517c09e3bf7046ccba6ed25b08e3460e
0506 19:56:41.894176 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.894348 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4207838}
I20240506 19:56:41.894551 13730 raft_consensus.cc:504] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.895040 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 299) took 4209 ms. Trace:
I20240506 19:56:41.895151 13242 rpcz_store.cc:276] 0506 19:56:37.685807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.688406 (+ 2599us) service_pool.cc:225] Handling call
0506 19:56:41.894855 (+4206449us) ts_tablet_manager.cc:1120] Deleting tablet 4acde41b2c03440a89243aae008779e1
0506 19:56:41.894866 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.894977 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4206198}
I20240506 19:56:41.895375 13730 leader_election.cc:288] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.895463 13249 consensus_queue.cc:260] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.895551 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 298) took 4209 ms. Trace:
I20240506 19:56:41.895659 13242 rpcz_store.cc:276] 0506 19:56:37.685698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.689353 (+ 3655us) service_pool.cc:225] Handling call
0506 19:56:41.895391 (+4206038us) ts_tablet_manager.cc:1120] Deleting tablet 5988b89cb52e4f858b52862138bed025
0506 19:56:41.895403 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.895493 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4205839}
I20240506 19:56:41.896452 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca394b7f53ce445e8778dd25d0cf25c7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.897322 13249 ts_tablet_manager.cc:1398] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:41.898643 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d57249f30a44dab3384092ec20b515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:41.899349 13057 leader_election.cc:341] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.906013 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0f28c1f55fd14aca86cb00e43b3a6b76
I20240506 19:56:41.907369 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:41.907685 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee59a76d52fc48d7bcf3256c359d3c02" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:41.908310 12922 leader_election.cc:341] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.908540 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a568db06324a4fd1b58a29ef0831eb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.917632 13791 tablet.cc:1722] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:41.898065 12810 tablet_service.cc:1467] Processing CreateTablet for tablet a568db06324a4fd1b58a29ef0831eb71 (DEFAULT_TABLE table=default.Table_46aaa890b4ce4d02919bab7b358c7a79 [id=5bf872e62221484ca77c5d51ae682edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.919193 13245 tablet_bootstrap.cc:492] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:41.919413 13245 ts_tablet_manager.cc:1367] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.337s user 0.041s sys 0.032s
I20240506 19:56:41.920434 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a568db06324a4fd1b58a29ef0831eb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.920254 13245 raft_consensus.cc:348] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.920553 13245 raft_consensus.cc:374] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.920647 13245 raft_consensus.cc:729] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.921059 13245 consensus_queue.cc:260] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.921304 13730 raft_consensus.cc:482] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.922060 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 55d9f32618b64ffa92b5a868081c36a6 (DEFAULT_TABLE table=default.Table_df0ba23556a64063af810291bf81830e [id=b0c9bae470ea4be1a475a6d536632412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.922188 13730 raft_consensus.cc:504] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.922884 13730 leader_election.cc:288] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:41.922966 13244 tablet_bootstrap.cc:492] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:41.923709 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d57249f30a44dab3384092ec20b515" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:41.924320 13057 leader_election.cc:341] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.922837 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee59a76d52fc48d7bcf3256c359d3c02" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:41.926518 12921 leader_election.cc:341] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.926666 12921 leader_election.cc:302] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.927172 13629 raft_consensus.cc:2726] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.933737 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 5b375490272a4dec9a743f6458f6d6de (DEFAULT_TABLE table=default.Table_90d5ee072fbd4121ae52144e32010742 [id=c749aef4b9e0450fada39042d70c1917]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.933140 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca394b7f53ce445e8778dd25d0cf25c7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:41.935575 13058 leader_election.cc:341] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:41.935766 13058 leader_election.cc:302] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:41.935933 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d9f32618b64ffa92b5a868081c36a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.936059 13781 raft_consensus.cc:2726] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:41.943205 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet bc85233b180f4b30a86bb65c86af684e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.943392 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb9c05d703f417780e77359e19e542e (DEFAULT_TABLE table=default.Table_c4128d96524a4c20b2f9a7ceedccfe96 [id=e6b450dbd2dd45308a4309aea08be26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.943775 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a8531a1793404699532295164a3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.943871 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb9c05d703f417780e77359e19e542e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.944384 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:41.946015 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e75fdc7de2104d288d8d6bf0353108f3" } new_table_name: "default.Table_200204bb39c0432081665c45b4aa9eae" modify_external_catalogs: true
I20240506 19:56:41.946202 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4ac15d4e3f6b4719b8b728be5474dd63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.946805 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ac15d4e3f6b4719b8b728be5474dd63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.948123 13716 raft_consensus.cc:482] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:41.948300 13716 raft_consensus.cc:504] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.948951 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_373183be2430417d8304f5e10cc72a57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.949066 13716 leader_election.cc:288] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:41.949481 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_373183be2430417d8304f5e10cc72a57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.949877 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_574f459d109b430ea8c288af9280052f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.950352 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_574f459d109b430ea8c288af9280052f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.950887 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 2b949c48806d45c3a74efbc8503c0bdb (DEFAULT_TABLE table=default.Table_9bca56a380f341c28cf5c3f683024098 [id=798e81e44a5a46ec962f48302ab404a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.951311 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 2751074a54ed4bcea6979e6ad11c2efe from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:41.951367 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b949c48806d45c3a74efbc8503c0bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.951906 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_670635380d484804bfa6a7c94116e631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.952370 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_670635380d484804bfa6a7c94116e631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.953471 13246 ts_tablet_manager.cc:1888] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:41.953437 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_90d5ee072fbd4121ae52144e32010742" } new_table_name: "default.Table_ab5b8e26274d4c17aafc75dabab5539a" modify_external_catalogs: true
I20240506 19:56:41.953595 13246 log.cc:1196] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d6c656e16c1045dea79bc23b64fdd5e8
I20240506 19:56:41.953980 13246 ts_tablet_manager.cc:1909] T d6c656e16c1045dea79bc23b64fdd5e8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:41.954599 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_195d90ff34ff4ca6b6282483c9691f47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.955082 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_195d90ff34ff4ca6b6282483c9691f47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.956137 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_93498ac3d84b4de4af0e703aa2b03878" } new_table_name: "default.Table_1a5485067c1a4e529059f5434babbdc7" modify_external_catalogs: true
I20240506 19:56:41.957057 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4 (DEFAULT_TABLE table=default.Table_57c82ed0d1044d20b72a0e9341b299c5 [id=ea7a610430e34ca2b3ab4868800722d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.957717 13245 ts_tablet_manager.cc:1398] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240506 19:56:41.957890 13244 ts_tablet_manager.cc:1367] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.309s user 0.002s sys 0.002s
I20240506 19:56:41.958113 13245 tablet_bootstrap.cc:492] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:41.887696 13243 tablet_replica.cc:330] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:41.958470 13243 raft_consensus.cc:2229] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.958837 13243 raft_consensus.cc:2258] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.958798 13244 raft_consensus.cc:348] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.959048 13244 raft_consensus.cc:374] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:41.959124 13244 raft_consensus.cc:729] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:41.959378 13244 consensus_queue.cc:260] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.959466 13243 ts_tablet_manager.cc:1875] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.959925 13244 ts_tablet_manager.cc:1398] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:41.960235 13244 tablet_bootstrap.cc:492] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:41.960122 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dafad97b193f40038dfd516c93b79e96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:41.960637 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dafad97b193f40038dfd516c93b79e96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.961932 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 028055b9df1f4667bd2e54909690e327 (DEFAULT_TABLE table=default.Table_51c8c97056294104afce55e84e26a550 [id=4a72c498da064f31a0c5437f8e0d1048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.963404 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 744864043f9347e0a13f8c2a7d6b147f (DEFAULT_TABLE table=default.Table_f7049cb5beca4e30b1f0a315789a1e1e [id=079ee8d63b57495c8432a5dc9e100947]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.963860 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744864043f9347e0a13f8c2a7d6b147f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.965987 13249 tablet_bootstrap.cc:492] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:41.966293 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d6c656e16c1045dea79bc23b64fdd5e8 (table default.Table_c905679471c7485fa59180f110b5ec1f [id=c4769761819944109eebf88cb916cb27]) successfully deleted
I20240506 19:56:41.966362 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4ad903e75547fa829b82cc722fef1e (DEFAULT_TABLE table=default.Table_f48001b944b84606b9e9e95a00e8a6a7 [id=e24a1c603b7e4f6aabcbeb5dd2df1f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.966753 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4ad903e75547fa829b82cc722fef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.967537 13249 tablet_bootstrap.cc:654] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.969455 13246 tablet_replica.cc:330] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:41.969697 13246 raft_consensus.cc:2229] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:41.969875 13246 raft_consensus.cc:2258] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:41.970453 13246 ts_tablet_manager.cc:1875] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:41.970957 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.974367 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 44ce8749846847108edf5f554139cbd9 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.974813 13245 tablet_bootstrap.cc:654] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:41.975169 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ce8749846847108edf5f554139cbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.976074 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028055b9df1f4667bd2e54909690e327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.976190 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502b423bd1040539fb1a3ebbece3c1c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:41.976437 12986 raft_consensus.cc:2454] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:41.976490 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 744864043f9347e0a13f8c2a7d6b147f (DEFAULT_TABLE table=default.Table_f7049cb5beca4e30b1f0a315789a1e1e [id=079ee8d63b57495c8432a5dc9e100947]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.976842 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744864043f9347e0a13f8c2a7d6b147f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.978219 12781 leader_election.cc:302] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
W20240506 19:56:41.978731 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60d1f01c0edb4f6b8a621c1643ca1afb consensus state has no leader
I20240506 19:56:41.979346 13716 raft_consensus.cc:2781] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:41.979434 13716 raft_consensus.cc:482] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:41.979300 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_11b0e013ac1943e3ad3061b8a451c8d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:41.979534 13716 raft_consensus.cc:3037] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:41.979923 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11b0e013ac1943e3ad3061b8a451c8d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:41.981627 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4ad903e75547fa829b82cc722fef1e (DEFAULT_TABLE table=default.Table_f48001b944b84606b9e9e95a00e8a6a7 [id=e24a1c603b7e4f6aabcbeb5dd2df1f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.982108 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4ad903e75547fa829b82cc722fef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.982512 13716 raft_consensus.cc:504] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:41.983733 13716 leader_election.cc:288] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:41.984794 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 028055b9df1f4667bd2e54909690e327 (DEFAULT_TABLE table=default.Table_51c8c97056294104afce55e84e26a550 [id=4a72c498da064f31a0c5437f8e0d1048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.985267 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bcd38427a79c48d1badd4f2536ff238f" } modify_external_catalogs: true
I20240506 19:56:41.985483 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bcd38427a79c48d1badd4f2536ff238f" } modify_external_catalogs: true
I20240506 19:56:41.985795 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2502b423bd1040539fb1a3ebbece3c1c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:41.986589 12986 raft_consensus.cc:3037] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:41.986858 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5a443c0d35741b9966b32a7808a97e2 consensus state has no leader
W20240506 19:56:41.987305 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b375490272a4dec9a743f6458f6d6de consensus state has no leader
I20240506 19:56:41.989629 12986 raft_consensus.cc:2454] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:56:41.992051 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 2b949c48806d45c3a74efbc8503c0bdb (DEFAULT_TABLE table=default.Table_9bca56a380f341c28cf5c3f683024098 [id=798e81e44a5a46ec962f48302ab404a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.992182 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 5b375490272a4dec9a743f6458f6d6de (DEFAULT_TABLE table=default.Table_90d5ee072fbd4121ae52144e32010742 [id=c749aef4b9e0450fada39042d70c1917]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:41.992585 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b375490272a4dec9a743f6458f6d6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:41.996466 13243 ts_tablet_manager.cc:1888] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.001417 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028055b9df1f4667bd2e54909690e327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.003484 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b949c48806d45c3a74efbc8503c0bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.007573 13243 log.cc:1196] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/6d8a9ecf47d7474c82758417449d940c
I20240506 19:56:42.008000 13243 ts_tablet_manager.cc:1909] T 6d8a9ecf47d7474c82758417449d940c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.008275 13245 tablet_bootstrap.cc:492] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:42.008427 13245 ts_tablet_manager.cc:1367] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240506 19:56:42.008870 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 355) took 3030 ms. Trace:
I20240506 19:56:42.008994 13243 rpcz_store.cc:276] 0506 19:56:38.978466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.981360 (+ 2894us) service_pool.cc:225] Handling call
0506 19:56:41.887591 (+2906231us) ts_tablet_manager.cc:1120] Deleting tablet 6d8a9ecf47d7474c82758417449d940c
0506 19:56:41.887605 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:41.993091 (+105486us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:41.996341 (+ 3250us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.008821 (+ 12480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2905989}
I20240506 19:56:42.009449 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b375490272a4dec9a743f6458f6d6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.009440 13245 raft_consensus.cc:348] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.009653 13245 raft_consensus.cc:374] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.010195 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 6d8a9ecf47d7474c82758417449d940c (table default.Table_8de44207890c4ceb8542f805a6f9b8da [id=70fc9e614eab4b11baeefbbf7127f540]) successfully deleted
I20240506 19:56:42.017031 13245 raft_consensus.cc:729] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.017378 13245 consensus_queue.cc:260] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.018558 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a80834187f874661b611b43065eee943 (table default.Table_2b51a8ac571347a896b8741066c330c4 [id=c18e335476d443f9949f5f38fa4550ae]) successfully deleted
W20240506 19:56:42.018828 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 517c09e3bf7046ccba6ed25b08e3460e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 517c09e3bf7046ccba6ed25b08e3460e already in progress: creating tablet
W20240506 19:56:42.019079 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4acde41b2c03440a89243aae008779e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4acde41b2c03440a89243aae008779e1 already in progress: creating tablet
I20240506 19:56:42.019034 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb9c05d703f417780e77359e19e542e (DEFAULT_TABLE table=default.Table_c4128d96524a4c20b2f9a7ceedccfe96 [id=e6b450dbd2dd45308a4309aea08be26d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:42.019511 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5988b89cb52e4f858b52862138bed025 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5988b89cb52e4f858b52862138bed025 already in progress: creating tablet
I20240506 19:56:42.020351 13245 ts_tablet_manager.cc:1398] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240506 19:56:42.021152 13245 tablet_bootstrap.cc:492] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:42.021656 13243 tablet_replica.cc:330] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.021894 13243 raft_consensus.cc:2229] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.022257 13243 raft_consensus.cc:2258] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.022841 13243 ts_tablet_manager.cc:1875] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.023432 13781 raft_consensus.cc:482] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.023586 13781 raft_consensus.cc:504] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.024766 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee59a76d52fc48d7bcf3256c359d3c02" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:42.024941 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb9c05d703f417780e77359e19e542e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.025066 12986 raft_consensus.cc:2454] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:42.025926 13057 leader_election.cc:302] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:42.026991 13781 leader_election.cc:288] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:42.027494 13730 raft_consensus.cc:2781] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:42.027601 13730 raft_consensus.cc:482] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:42.030195 12781 leader_election.cc:302] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:42.025465 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d57249f30a44dab3384092ec20b515" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:42.032255 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.032438 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a8531a1793404699532295164a3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.033353 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee59a76d52fc48d7bcf3256c359d3c02" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:42.033687 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet bc85233b180f4b30a86bb65c86af684e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.037312 13730 raft_consensus.cc:3037] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:42.038619 13246 ts_tablet_manager.cc:1888] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:42.044158 13058 leader_election.cc:341] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.044473 13058 leader_election.cc:302] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:42.044644 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet f3d57249f30a44dab3384092ec20b515 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.044759 13781 raft_consensus.cc:2726] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.045468 13246 log.cc:1196] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:42.045809 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9cd8092716234d9391b0d018843e289f" } new_table_name: "default.Table_5140ce0f4218408182d418c041a2535c" modify_external_catalogs: true
I20240506 19:56:42.046952 13730 raft_consensus.cc:504] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.049360 13246 ts_tablet_manager.cc:1909] T 2751074a54ed4bcea6979e6ad11c2efe P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:42.049561 13058 leader_election.cc:341] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.047367 13245 tablet_bootstrap.cc:654] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.050318 13730 leader_election.cc:288] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:42.050582 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee59a76d52fc48d7bcf3256c359d3c02" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:42.050801 12986 raft_consensus.cc:3037] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:42.052204 12783 leader_election.cc:341] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2502b423bd1040539fb1a3ebbece3c1c
W20240506 19:56:42.052570 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3e748d2c32a4b09abc872fed318e643 consensus state has no leader
W20240506 19:56:42.052736 12783 leader_election.cc:341] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2502b423bd1040539fb1a3ebbece3c1c
I20240506 19:56:42.053766 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bb8517f5ef13428c9dcfc8c72d0cab0e" } modify_external_catalogs: true
I20240506 19:56:42.054402 13244 tablet_bootstrap.cc:654] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.054518 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bb8517f5ef13428c9dcfc8c72d0cab0e" } modify_external_catalogs: true
W20240506 19:56:42.054682 12783 leader_election.cc:341] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 393073bca2a04d15b3e583700262f3e0
I20240506 19:56:42.054827 12783 leader_election.cc:302] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:42.055497 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet bff7050bbc2e47cf823d25ec10b89718 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.055536 12986 raft_consensus.cc:2454] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:42.055985 13057 leader_election.cc:302] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:42.056346 13730 raft_consensus.cc:2781] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:42.056984 13730 raft_consensus.cc:686] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:42.057399 13730 consensus_queue.cc:237] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.058018 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2751074a54ed4bcea6979e6ad11c2efe (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]) successfully deleted
W20240506 19:56:42.059479 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d6e54543d1544b1c8cff3e890e787337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6e54543d1544b1c8cff3e890e787337 already in progress: creating tablet
W20240506 19:56:42.060055 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8d09eafa82a0423a8358958e78f728e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d09eafa82a0423a8358958e78f728e2 already in progress: creating tablet
W20240506 19:56:42.060529 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a1160ab972124630a9be7abeefb4aa56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1160ab972124630a9be7abeefb4aa56 already in progress: creating tablet
W20240506 19:56:42.061002 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2a7c61244620487bbbac26dcdbf8a38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7c61244620487bbbac26dcdbf8a38c already in progress: creating tablet
W20240506 19:56:42.061491 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 234e4f3439e84b5fbb2f7bab1dbd0822 already in progress: creating tablet
W20240506 19:56:42.061939 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a9e90b93023e40b3bafbddf7bbcc10b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e90b93023e40b3bafbddf7bbcc10b2 already in progress: creating tablet
I20240506 19:56:42.054107 13246 tablet_replica.cc:330] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:42.067878 13246 raft_consensus.cc:2229] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.067919 13243 ts_tablet_manager.cc:1888] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.068071 13243 log.cc:1196] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2502b423bd1040539fb1a3ebbece3c1c
I20240506 19:56:42.068109 13246 raft_consensus.cc:2258] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.068471 13243 ts_tablet_manager.cc:1909] T 2502b423bd1040539fb1a3ebbece3c1c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.068702 13246 ts_tablet_manager.cc:1875] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.070945 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee59a76d52fc48d7bcf3256c359d3c02" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:42.073511 13058 leader_election.cc:341] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.081526 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 361) took 2855 ms. Trace:
I20240506 19:56:42.081772 13243 rpcz_store.cc:276] 0506 19:56:39.214200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.214257 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:42.021571 (+2807314us) ts_tablet_manager.cc:1120] Deleting tablet 2502b423bd1040539fb1a3ebbece3c1c
0506 19:56:42.021584 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.030532 (+ 8948us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.067777 (+ 37245us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.069242 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2807036}
I20240506 19:56:42.082378 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2502b423bd1040539fb1a3ebbece3c1c (table default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]) successfully deleted
I20240506 19:56:42.082585 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 360) took 2868 ms. Trace:
I20240506 19:56:42.082747 13243 rpcz_store.cc:276] 0506 19:56:39.214066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.221061 (+ 6995us) service_pool.cc:225] Handling call
0506 19:56:42.082359 (+2861298us) ts_tablet_manager.cc:1120] Deleting tablet 1615f6167aae4dc096b9f563ab5e6215
0506 19:56:42.082376 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.082525 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2861048}
W20240506 19:56:42.083377 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1615f6167aae4dc096b9f563ab5e6215 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1615f6167aae4dc096b9f563ab5e6215 already in progress: creating tablet
I20240506 19:56:42.083509 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 364) took 2844 ms. Trace:
I20240506 19:56:42.083595 13243 rpcz_store.cc:276] 0506 19:56:39.238549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.238597 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:42.083332 (+2844735us) ts_tablet_manager.cc:1120] Deleting tablet 85bc7cfd006f4d4ca95541dab6873d32
0506 19:56:42.083341 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.083459 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2844511}
W20240506 19:56:42.084039 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 85bc7cfd006f4d4ca95541dab6873d32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85bc7cfd006f4d4ca95541dab6873d32 already in progress: creating tablet
I20240506 19:56:42.084112 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 363) took 2845 ms. Trace:
I20240506 19:56:42.084245 13242 tablet_replica.cc:330] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:42.084197 13243 rpcz_store.cc:276] 0506 19:56:39.238411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.238458 (+ 47us) service_pool.cc:225] Handling call
0506 19:56:42.083984 (+2845526us) ts_tablet_manager.cc:1120] Deleting tablet f8951bf4873e4382bcd42a52368fb376
0506 19:56:42.083991 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.084061 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2842822}
W20240506 19:56:42.084506 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f8951bf4873e4382bcd42a52368fb376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8951bf4873e4382bcd42a52368fb376 already in progress: creating tablet
I20240506 19:56:42.086089 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 362) took 2847 ms. Trace:
I20240506 19:56:42.086284 13243 rpcz_store.cc:276] 0506 19:56:39.238239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.238311 (+ 72us) service_pool.cc:225] Handling call
0506 19:56:42.085895 (+2847584us) ts_tablet_manager.cc:1120] Deleting tablet d7be9788df8e47d285bd932d078c6a8b
0506 19:56:42.085907 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.086014 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2842712}
W20240506 19:56:42.086800 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d7be9788df8e47d285bd932d078c6a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7be9788df8e47d285bd932d078c6a8b already in progress: creating tablet
I20240506 19:56:42.086906 13243 tablet_replica.cc:330] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.087096 13243 raft_consensus.cc:2229] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.087396 13243 raft_consensus.cc:2258] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.088117 13243 ts_tablet_manager.cc:1875] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:42.090248 12506 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 5b375490272a4dec9a743f6458f6d6de (table default.Table_ab5b8e26274d4c17aafc75dabab5539a [id=c749aef4b9e0450fada39042d70c1917], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5b375490272a4dec9a743f6458f6d6de consensus state has no leader
I20240506 19:56:42.095458 13243 ts_tablet_manager.cc:1888] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.086031 13716 raft_consensus.cc:2781] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:42.099745 13243 log.cc:1196] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/27b24fa4663244a88dac9ea99fe38f3a
I20240506 19:56:42.101243 13243 ts_tablet_manager.cc:1909] T 27b24fa4663244a88dac9ea99fe38f3a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.105525 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 366) took 2760 ms. Trace:
I20240506 19:56:42.105784 13243 rpcz_store.cc:276] 0506 19:56:39.344707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.345837 (+ 1130us) service_pool.cc:225] Handling call
0506 19:56:42.086803 (+2740966us) ts_tablet_manager.cc:1120] Deleting tablet 27b24fa4663244a88dac9ea99fe38f3a
0506 19:56:42.086815 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.091703 (+ 4888us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.095323 (+ 3620us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.105460 (+ 10137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2740705}
I20240506 19:56:42.106503 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 27b24fa4663244a88dac9ea99fe38f3a (table default.Table_1f7a4e714f4a4412a5843504c95b87d7 [id=9844f952118d4225a5a7489527a6a703]) successfully deleted
I20240506 19:56:42.106618 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 388) took 2389 ms. Trace:
I20240506 19:56:42.106727 13243 rpcz_store.cc:276] 0506 19:56:39.716707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.896900 (+180193us) service_pool.cc:225] Handling call
0506 19:56:42.106429 (+2209529us) ts_tablet_manager.cc:1120] Deleting tablet 6d04a1c5ee7b4d529fa2f4da4945ff09
0506 19:56:42.106441 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.106587 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834443}
I20240506 19:56:42.107273 13243 tablet_replica.cc:330] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:42.107246 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 already in progress: creating tablet
I20240506 19:56:42.107559 13243 raft_consensus.cc:2229] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.107805 13243 raft_consensus.cc:2258] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.109273 13629 raft_consensus.cc:482] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.109544 13629 raft_consensus.cc:504] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.110406 13629 leader_election.cc:288] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.110956 13243 ts_tablet_manager.cc:1875] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.111531 13242 raft_consensus.cc:2229] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.093588 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 44ce8749846847108edf5f554139cbd9 (DEFAULT_TABLE table=default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.114955 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c656e16c1045dea79bc23b64fdd5e8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:42.116276 13242 raft_consensus.cc:2258] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.117604 13242 ts_tablet_manager.cc:1875] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:42.116097 12921 leader_election.cc:341] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.129664 13243 ts_tablet_manager.cc:1888] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:42.129985 13243 log.cc:1196] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/320e714ed6b64f2d84770e7182e4103a
I20240506 19:56:42.130599 13243 ts_tablet_manager.cc:1909] T 320e714ed6b64f2d84770e7182e4103a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.131508 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 389) took 2403 ms. Trace:
I20240506 19:56:42.131795 13243 rpcz_store.cc:276] 0506 19:56:39.727718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.897499 (+169781us) service_pool.cc:225] Handling call
0506 19:56:42.107212 (+2209713us) ts_tablet_manager.cc:1120] Deleting tablet 320e714ed6b64f2d84770e7182e4103a
0506 19:56:42.107220 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.126230 (+ 19010us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.129493 (+ 3263us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.131430 (+ 1937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834560}
I20240506 19:56:42.132522 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 320e714ed6b64f2d84770e7182e4103a (table default.Table_8be586b55bf2436698bfeb2f94fb465d [id=090dfa9462424be1b846e1cb05569156]) successfully deleted
I20240506 19:56:42.132635 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 391) took 2404 ms. Trace:
I20240506 19:56:42.132735 13243 rpcz_store.cc:276] 0506 19:56:39.727953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.901502 (+173549us) service_pool.cc:225] Handling call
0506 19:56:42.132465 (+2230963us) ts_tablet_manager.cc:1120] Deleting tablet 997ecba7fbae41178a25bf7af4d152e1
0506 19:56:42.132475 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.132578 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1854060}
W20240506 19:56:42.136508 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 997ecba7fbae41178a25bf7af4d152e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 997ecba7fbae41178a25bf7af4d152e1 already in progress: creating tablet
I20240506 19:56:42.137174 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 390) took 2409 ms. Trace:
I20240506 19:56:42.137284 13243 rpcz_store.cc:276] 0506 19:56:39.727840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.900435 (+172595us) service_pool.cc:225] Handling call
0506 19:56:42.136476 (+2236041us) ts_tablet_manager.cc:1120] Deleting tablet 4095f9fc28ff48f68064ec8318ff7b51
0506 19:56:42.136485 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.137107 (+ 622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1857750}
W20240506 19:56:42.137746 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4095f9fc28ff48f68064ec8318ff7b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4095f9fc28ff48f68064ec8318ff7b51 already in progress: creating tablet
I20240506 19:56:42.138216 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 396) took 1818 ms. Trace:
I20240506 19:56:42.138314 13243 rpcz_store.cc:276] 0506 19:56:40.319587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.319798 (+ 211us) service_pool.cc:225] Handling call
0506 19:56:42.138063 (+1818265us) ts_tablet_manager.cc:1120] Deleting tablet 2519c03738864f6ab5dcfab7bc3fc8aa
0506 19:56:42.138073 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.138170 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1817998}
W20240506 19:56:42.138731 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2519c03738864f6ab5dcfab7bc3fc8aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2519c03738864f6ab5dcfab7bc3fc8aa already in progress: creating tablet
I20240506 19:56:42.138784 13243 tablet_replica.cc:330] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.138983 13243 raft_consensus.cc:2229] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:42.139434 13243 raft_consensus.cc:2258] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.142081 13243 ts_tablet_manager.cc:1875] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.144658 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fa840bd001a04582ac72848588e27126" } modify_external_catalogs: true
I20240506 19:56:42.144899 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fa840bd001a04582ac72848588e27126" } modify_external_catalogs: true
I20240506 19:56:42.145540 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_012b93e45a4945ee964f19ccd6bbfa0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.146127 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_012b93e45a4945ee964f19ccd6bbfa0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.147454 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5553b476197648cd9965dd87bbabb42f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.147969 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5553b476197648cd9965dd87bbabb42f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.149395 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_942245e6c2a94a96af9206be9765c407"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.149835 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_942245e6c2a94a96af9206be9765c407 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.150346 13243 ts_tablet_manager.cc:1888] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240506 19:56:42.150491 13243 log.cc:1196] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/6156c6f582b747de973b357b6d071a46
I20240506 19:56:42.150871 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bf189fc046d248199cdc55fe6c701660" } new_table_name: "default.Table_3dea609720914447924d3591bfed24f6" modify_external_catalogs: true
I20240506 19:56:42.150982 13243 ts_tablet_manager.cc:1909] T 6156c6f582b747de973b357b6d071a46 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.151865 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 397) took 1396 ms. Trace:
I20240506 19:56:42.151975 13243 rpcz_store.cc:276] 0506 19:56:40.755302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.755356 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:42.138706 (+1383350us) ts_tablet_manager.cc:1120] Deleting tablet 6156c6f582b747de973b357b6d071a46
0506 19:56:42.138714 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.146894 (+ 8180us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.150216 (+ 3322us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.151775 (+ 1559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380055}
I20240506 19:56:42.152515 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_44e570af6caa4a749207a8c828639acf [id=ee08d3829f304e16b5b8994259bb11b0]) successfully deleted
I20240506 19:56:42.152918 13243 tablet_replica.cc:330] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.153210 13243 raft_consensus.cc:2229] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.153487 13243 raft_consensus.cc:2258] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.154220 13243 ts_tablet_manager.cc:1875] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.155784 12483 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu pid 12630
I20240506 19:56:42.157137 12694 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 19:56:42.161931 13243 ts_tablet_manager.cc:1888] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.162662 13243 log.cc:1196] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/30d819d1237f4798b4847fdf378cbc62
I20240506 19:56:42.163134 13243 ts_tablet_manager.cc:1909] T 30d819d1237f4798b4847fdf378cbc62 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.167057 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 398) took 1382 ms. Trace:
I20240506 19:56:42.167232 13243 rpcz_store.cc:276] 0506 19:56:40.784730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.784799 (+ 69us) service_pool.cc:225] Handling call
0506 19:56:42.152825 (+1368026us) ts_tablet_manager.cc:1120] Deleting tablet 30d819d1237f4798b4847fdf378cbc62
0506 19:56:42.152835 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.158066 (+ 5231us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.161816 (+ 3750us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.166995 (+ 5179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1367768}
I20240506 19:56:42.167670 13243 tablet_replica.cc:330] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.167901 13243 raft_consensus.cc:2229] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:42.168259 13243 raft_consensus.cc:2258] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.168632 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 30d819d1237f4798b4847fdf378cbc62 (table default.Table_9d121b9e240b43cfbfea349bf45ef3a7 [id=332ee4f1670e4ddba0deb90d49e7b964]) successfully deleted
I20240506 19:56:42.168829 13243 ts_tablet_manager.cc:1875] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.176108 13243 ts_tablet_manager.cc:1888] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:42.176368 13243 log.cc:1196] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/5dc0ac37f67942fc8a31d8d43c839246
I20240506 19:56:42.177023 13243 ts_tablet_manager.cc:1909] T 5dc0ac37f67942fc8a31d8d43c839246 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.178130 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 400) took 1301 ms. Trace:
I20240506 19:56:42.178371 13243 rpcz_store.cc:276] 0506 19:56:40.876855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.876917 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:42.167586 (+1290669us) ts_tablet_manager.cc:1120] Deleting tablet 5dc0ac37f67942fc8a31d8d43c839246
0506 19:56:42.167596 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.172708 (+ 5112us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.175976 (+ 3268us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.178070 (+ 2094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290300}
I20240506 19:56:42.179126 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 5dc0ac37f67942fc8a31d8d43c839246 (table default.Table_eba34a76ef98415ea1596d0f5bd50b7f [id=7466cd9f5b1c40ae857ea1a714a84148]) successfully deleted
I20240506 19:56:42.179759 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 402) took 1264 ms. Trace:
I20240506 19:56:42.179917 13243 rpcz_store.cc:276] 0506 19:56:40.915718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.915781 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:42.179553 (+1263772us) ts_tablet_manager.cc:1120] Deleting tablet 034d6309de0740d0a5806b3847e334cb
0506 19:56:42.179567 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.179721 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263501}
I20240506 19:56:42.180706 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 401) took 1265 ms. Trace:
I20240506 19:56:42.181725 13793 tablet.cc:1722] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:42.191150 13243 rpcz_store.cc:276] 0506 19:56:40.915513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.916105 (+ 592us) service_pool.cc:225] Handling call
0506 19:56:42.180591 (+1264486us) ts_tablet_manager.cc:1120] Deleting tablet f8ffb88e0a0c4125b04f4d8d328da3f5
0506 19:56:42.180598 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.180669 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264239}
W20240506 19:56:42.191776 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8ffb88e0a0c4125b04f4d8d328da3f5 already in progress: creating tablet
I20240506 19:56:42.192250 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 403) took 1217 ms. Trace:
I20240506 19:56:42.192421 13243 rpcz_store.cc:276] 0506 19:56:40.975065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.975358 (+ 293us) service_pool.cc:225] Handling call
0506 19:56:42.192111 (+1216753us) ts_tablet_manager.cc:1120] Deleting tablet bece304e0ba048ecbb2d92aa71fdf707
0506 19:56:42.192121 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.192218 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216021}
W20240506 19:56:42.192916 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bece304e0ba048ecbb2d92aa71fdf707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bece304e0ba048ecbb2d92aa71fdf707 already in progress: creating tablet
I20240506 19:56:42.193490 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 404) took 1218 ms. Trace:
I20240506 19:56:42.193691 13243 rpcz_store.cc:276] 0506 19:56:40.975166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.976127 (+ 961us) service_pool.cc:225] Handling call
0506 19:56:42.193239 (+1217112us) ts_tablet_manager.cc:1120] Deleting tablet c1ff7a3638004012b08c973192c055a5
0506 19:56:42.193251 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.193445 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216936}
W20240506 19:56:42.194243 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c1ff7a3638004012b08c973192c055a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1ff7a3638004012b08c973192c055a5 already in progress: creating tablet
I20240506 19:56:42.194835 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 405) took 1219 ms. Trace:
I20240506 19:56:42.195019 13243 rpcz_store.cc:276] 0506 19:56:40.975238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.976318 (+ 1080us) service_pool.cc:225] Handling call
0506 19:56:42.194612 (+1218294us) ts_tablet_manager.cc:1120] Deleting tablet 373e04e93a5543f9934077006eaf2ba9
0506 19:56:42.194624 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.194791 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1218153}
W20240506 19:56:42.195487 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 373e04e93a5543f9934077006eaf2ba9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 373e04e93a5543f9934077006eaf2ba9 already in progress: creating tablet
I20240506 19:56:42.195786 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_bcd38427a79c48d1badd4f2536ff238f [id=f23583e607a4419986d06a38c45bbdf5]). Expected version 2 got 0
I20240506 19:56:42.195894 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 406) took 1220 ms. Trace:
I20240506 19:56:42.196521 13243 rpcz_store.cc:276] 0506 19:56:40.975317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.976474 (+ 1157us) service_pool.cc:225] Handling call
0506 19:56:42.195769 (+1219295us) ts_tablet_manager.cc:1120] Deleting tablet 1a6e0126d9854215a0e174a76c0a0391
0506 19:56:42.195777 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.195863 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219160}
W20240506 19:56:42.196940 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1a6e0126d9854215a0e174a76c0a0391 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a6e0126d9854215a0e174a76c0a0391 already in progress: creating tablet
I20240506 19:56:42.197240 13243 tablet_replica.cc:330] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.197607 13243 raft_consensus.cc:2229] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.197853 13243 raft_consensus.cc:2258] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.198611 13243 ts_tablet_manager.cc:1875] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.124759 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4 (DEFAULT_TABLE table=default.Table_57c82ed0d1044d20b72a0e9341b299c5 [id=ea7a610430e34ca2b3ab4868800722d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.133494 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ce8749846847108edf5f554139cbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.136853 13716 raft_consensus.cc:686] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:56:42.147362 13802 raft_consensus.cc:2726] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.187330 13807 tablet.cc:1722] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:42.206461 13243 ts_tablet_manager.cc:1888] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.206669 13243 log.cc:1196] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:42.207074 13243 ts_tablet_manager.cc:1909] T 72a3bed0eb234e21b890242e9d23a5b4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.207871 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 407) took 1188 ms. Trace:
I20240506 19:56:42.208025 13243 rpcz_store.cc:276] 0506 19:56:41.019021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.019340 (+ 319us) service_pool.cc:225] Handling call
0506 19:56:42.197183 (+1177843us) ts_tablet_manager.cc:1120] Deleting tablet 72a3bed0eb234e21b890242e9d23a5b4
0506 19:56:42.197191 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.202768 (+ 5577us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.206338 (+ 3570us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.207813 (+ 1475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1177592}
I20240506 19:56:42.208608 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 72a3bed0eb234e21b890242e9d23a5b4 (table default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]) successfully deleted
I20240506 19:56:42.208990 13243 tablet_replica.cc:330] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.209241 13243 raft_consensus.cc:2229] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.209543 13243 raft_consensus.cc:2258] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.210366 13243 ts_tablet_manager.cc:1875] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.217347 13243 ts_tablet_manager.cc:1888] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.217623 13243 log.cc:1196] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:42.218154 13243 ts_tablet_manager.cc:1909] T 8f66f1e46cce47639fe644fb4ed2e5c0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.219094 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 408) took 1199 ms. Trace:
I20240506 19:56:42.219276 13243 rpcz_store.cc:276] 0506 19:56:41.019194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.019629 (+ 435us) service_pool.cc:225] Handling call
0506 19:56:42.208906 (+1189277us) ts_tablet_manager.cc:1120] Deleting tablet 8f66f1e46cce47639fe644fb4ed2e5c0
0506 19:56:42.208916 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.214159 (+ 5243us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.217209 (+ 3050us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.219050 (+ 1841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189072}
I20240506 19:56:42.219544 13249 tablet_bootstrap.cc:492] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.219743 13249 ts_tablet_manager.cc:1367] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.254s user 0.033s sys 0.025s
I20240506 19:56:42.219861 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 8f66f1e46cce47639fe644fb4ed2e5c0 (table default.Table_ca61f3a9aefa4fba8fea32d717de3fb1 [id=bf75c039ad324e50a2ccb889439e2b4c]) successfully deleted
I20240506 19:56:42.220279 13243 tablet_replica.cc:330] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.220548 13243 raft_consensus.cc:2229] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:42.220654 13249 raft_consensus.cc:348] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.220868 13249 raft_consensus.cc:374] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.220969 13249 raft_consensus.cc:729] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.221143 13243 pending_rounds.cc:62] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:42.221263 13243 pending_rounds.cc:68] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744041825566720 op_type: NO_OP noop_request { }
I20240506 19:56:42.221472 13243 raft_consensus.cc:2866] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:42.221627 13243 raft_consensus.cc:2258] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.221545 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_eba34a76ef98415ea1596d0f5bd50b7f" } modify_external_catalogs: true
I20240506 19:56:42.221776 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_eba34a76ef98415ea1596d0f5bd50b7f" } modify_external_catalogs: true
I20240506 19:56:42.222198 13243 ts_tablet_manager.cc:1875] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.222772 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0ea05891ea5e48a5b2a179a245749dc7" } modify_external_catalogs: true
I20240506 19:56:42.222949 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0ea05891ea5e48a5b2a179a245749dc7" } modify_external_catalogs: true
I20240506 19:56:42.221216 13249 consensus_queue.cc:260] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.227394 13249 ts_tablet_manager.cc:1398] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:42.227835 13249 tablet_bootstrap.cc:492] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.229588 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]). Expected version 2 got 0
I20240506 19:56:42.231865 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet d5725b47e43244f0af881b07e145b3aa from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.232509 13249 tablet_bootstrap.cc:654] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.235836 13243 ts_tablet_manager.cc:1888] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:42.236049 13243 log.cc:1196] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:42.236611 13243 ts_tablet_manager.cc:1909] T 40ece0ad4df247eb9d380ed1c755ac99 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.237936 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 409) took 1218 ms. Trace:
I20240506 19:56:42.238121 13243 rpcz_store.cc:276] 0506 19:56:41.019289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.019859 (+ 570us) service_pool.cc:225] Handling call
0506 19:56:42.220206 (+1200347us) ts_tablet_manager.cc:1120] Deleting tablet 40ece0ad4df247eb9d380ed1c755ac99
0506 19:56:42.220215 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.225966 (+ 5751us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.235682 (+ 9716us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.237886 (+ 2204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200205}
I20240506 19:56:42.238734 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 40ece0ad4df247eb9d380ed1c755ac99 (table default.Table_d91e484ef0cc4adb86af57d549a7e641 [id=9ad0be2302b94351876dba7a918a8d22]) successfully deleted
I20240506 19:56:42.239230 13243 tablet_replica.cc:330] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:42.239473 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 796628a0859c4511a8cb35e495dc8886 consensus state has no leader
I20240506 19:56:42.239490 13243 raft_consensus.cc:2229] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.239696 13243 raft_consensus.cc:2258] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.240547 13243 ts_tablet_manager.cc:1875] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.246963 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4cc2d777e8ad43d49ca9092206e8dc72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.247375 13249 tablet_bootstrap.cc:492] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.247539 13249 ts_tablet_manager.cc:1367] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.002s
W20240506 19:56:42.247737 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cc2d777e8ad43d49ca9092206e8dc72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.248363 13243 ts_tablet_manager.cc:1888] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.248504 13249 raft_consensus.cc:348] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.248742 13249 raft_consensus.cc:374] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.248903 13249 raft_consensus.cc:729] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.249147 13249 consensus_queue.cc:260] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.249797 13249 ts_tablet_manager.cc:1398] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:42.248513 13243 log.cc:1196] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b38453bf678748399339806d12993c88
I20240506 19:56:42.250627 13243 ts_tablet_manager.cc:1909] T b38453bf678748399339806d12993c88 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.251426 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 410) took 1197 ms. Trace:
I20240506 19:56:42.251562 13243 rpcz_store.cc:276] 0506 19:56:41.053552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.053752 (+ 200us) service_pool.cc:225] Handling call
0506 19:56:42.239145 (+1185393us) ts_tablet_manager.cc:1120] Deleting tablet b38453bf678748399339806d12993c88
0506 19:56:42.239158 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.244782 (+ 5624us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.248256 (+ 3474us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.251372 (+ 3116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166916}
I20240506 19:56:42.251966 13243 tablet_replica.cc:330] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.252200 13243 raft_consensus.cc:2229] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.252431 13243 raft_consensus.cc:2258] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.252441 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b38453bf678748399339806d12993c88 (table default.Table_62612e9478664e9d86a80129badd6c12 [id=8f671c29345e43b6ba8379f8182cba65]) successfully deleted
I20240506 19:56:42.253002 13243 ts_tablet_manager.cc:1875] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.250228 13249 tablet_bootstrap.cc:492] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.255141 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa
I20240506 19:56:42.255471 13249 tablet_bootstrap.cc:654] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.257400 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.264863 13243 ts_tablet_manager.cc:1888] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.265027 13243 log.cc:1196] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:42.265421 13243 ts_tablet_manager.cc:1909] T 2751074a54ed4bcea6979e6ad11c2efe P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.266230 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 411) took 1212 ms. Trace:
I20240506 19:56:42.266358 13243 rpcz_store.cc:276] 0506 19:56:41.053694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.053997 (+ 303us) service_pool.cc:225] Handling call
0506 19:56:42.251877 (+1197880us) ts_tablet_manager.cc:1120] Deleting tablet 2751074a54ed4bcea6979e6ad11c2efe
0506 19:56:42.251887 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.259273 (+ 7386us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.264723 (+ 5450us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.266193 (+ 1470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179576}
I20240506 19:56:42.266806 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 420) took 1030 ms. Trace:
I20240506 19:56:42.266893 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2751074a54ed4bcea6979e6ad11c2efe (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]) successfully deleted
I20240506 19:56:42.266925 13243 rpcz_store.cc:276] 0506 19:56:41.236104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.236594 (+ 490us) service_pool.cc:225] Handling call
0506 19:56:42.266658 (+1030064us) ts_tablet_manager.cc:1120] Deleting tablet a1160ab972124630a9be7abeefb4aa56
0506 19:56:42.266669 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.266776 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029812}
I20240506 19:56:42.267292 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 421) took 1031 ms. Trace:
W20240506 19:56:42.267398 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a1160ab972124630a9be7abeefb4aa56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1160ab972124630a9be7abeefb4aa56 already in progress: creating tablet
I20240506 19:56:42.267402 13243 rpcz_store.cc:276] 0506 19:56:41.236217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.236880 (+ 663us) service_pool.cc:225] Handling call
0506 19:56:42.267172 (+1030292us) ts_tablet_manager.cc:1120] Deleting tablet 8d09eafa82a0423a8358958e78f728e2
0506 19:56:42.267180 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.267262 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030118}
I20240506 19:56:42.267725 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 422) took 1031 ms. Trace:
I20240506 19:56:42.267839 13243 rpcz_store.cc:276] 0506 19:56:41.236328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.237083 (+ 755us) service_pool.cc:225] Handling call
0506 19:56:42.267614 (+1030531us) ts_tablet_manager.cc:1120] Deleting tablet 234e4f3439e84b5fbb2f7bab1dbd0822
0506 19:56:42.267622 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.267697 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030351}
W20240506 19:56:42.267983 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8d09eafa82a0423a8358958e78f728e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d09eafa82a0423a8358958e78f728e2 already in progress: creating tablet
I20240506 19:56:42.268218 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 423) took 1031 ms. Trace:
W20240506 19:56:42.268376 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 234e4f3439e84b5fbb2f7bab1dbd0822 already in progress: creating tablet
I20240506 19:56:42.268325 13243 rpcz_store.cc:276] 0506 19:56:41.236494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.237312 (+ 818us) service_pool.cc:225] Handling call
0506 19:56:42.268065 (+1030753us) ts_tablet_manager.cc:1120] Deleting tablet 2a7c61244620487bbbac26dcdbf8a38c
0506 19:56:42.268073 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.268177 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030561}
I20240506 19:56:42.268698 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 419) took 1032 ms. Trace:
I20240506 19:56:42.268808 13243 rpcz_store.cc:276] 0506 19:56:41.235901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.238132 (+ 2231us) service_pool.cc:225] Handling call
0506 19:56:42.268585 (+1030453us) ts_tablet_manager.cc:1120] Deleting tablet d6e54543d1544b1c8cff3e890e787337
0506 19:56:42.268594 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.268670 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030105}
W20240506 19:56:42.268927 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2a7c61244620487bbbac26dcdbf8a38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7c61244620487bbbac26dcdbf8a38c already in progress: creating tablet
I20240506 19:56:42.269189 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 425) took 1024 ms. Trace:
W20240506 19:56:42.269362 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d6e54543d1544b1c8cff3e890e787337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6e54543d1544b1c8cff3e890e787337 already in progress: creating tablet
I20240506 19:56:42.269315 13243 rpcz_store.cc:276] 0506 19:56:41.244762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.244824 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:42.269036 (+1024212us) ts_tablet_manager.cc:1120] Deleting tablet a9e90b93023e40b3bafbddf7bbcc10b2
0506 19:56:42.269045 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.269154 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023350}
I20240506 19:56:42.269601 13243 tablet_replica.cc:330] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:42.269770 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a9e90b93023e40b3bafbddf7bbcc10b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e90b93023e40b3bafbddf7bbcc10b2 already in progress: creating tablet
I20240506 19:56:42.269817 13243 raft_consensus.cc:2229] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.270010 13243 raft_consensus.cc:2258] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.270603 13243 ts_tablet_manager.cc:1875] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.274757 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:42.276371 12922 leader_election.cc:341] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d6c656e16c1045dea79bc23b64fdd5e8
I20240506 19:56:42.276530 12922 leader_election.cc:302] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:42.276980 13353 raft_consensus.cc:2726] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.282876 13353 raft_consensus.cc:482] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.283092 13353 raft_consensus.cc:504] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.283969 13353 leader_election.cc:288] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.284826 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:42.284960 13246 ts_tablet_manager.cc:1888] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.285086 13246 log.cc:1196] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1488328703024ae8983add9e4d6afc8f
I20240506 19:56:42.285096 13123 raft_consensus.cc:2454] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:42.285588 13246 ts_tablet_manager.cc:1909] T 1488328703024ae8983add9e4d6afc8f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:42.285835 12922 leader_election.cc:302] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:42.286192 13353 raft_consensus.cc:2781] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:42.286315 13353 raft_consensus.cc:482] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:42.286427 13353 raft_consensus.cc:3037] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:42.286976 13243 ts_tablet_manager.cc:1888] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.287144 13243 log.cc:1196] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/5988b89cb52e4f858b52862138bed025
I20240506 19:56:42.287155 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1488328703024ae8983add9e4d6afc8f (table default.Table_3501606fb8de42f68fd7894fa92e1a95 [id=3cabe6d9486d4b6fa610e1a880a5cdaa]) successfully deleted
I20240506 19:56:42.287561 13243 ts_tablet_manager.cc:1909] T 5988b89cb52e4f858b52862138bed025 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:42.287909 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2417a535329341d8bb9f8218770f890d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2417a535329341d8bb9f8218770f890d already in progress: creating tablet
I20240506 19:56:42.288640 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 5988b89cb52e4f858b52862138bed025 (table default.Table_7ec0af39e86c4ffeb0622785e6a817be [id=cd2382012b444dd09cd9ad1a50a20b4c]) successfully deleted
I20240506 19:56:42.288969 13243 tablet_replica.cc:330] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.289168 13243 raft_consensus.cc:2229] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:42.289211 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2417a535329341d8bb9f8218770f890d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2417a535329341d8bb9f8218770f890d already in progress: creating tablet
W20240506 19:56:42.289466 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 458314e4653748e48cf646c72373fc3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 458314e4653748e48cf646c72373fc3e already in progress: creating tablet
W20240506 19:56:42.289716 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b6fa6add23b8403ba5a0a65ac6752a5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fa6add23b8403ba5a0a65ac6752a5b already in progress: creating tablet
W20240506 19:56:42.289953 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ff3049a9fa8049e5b84230e5e6e8245c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3049a9fa8049e5b84230e5e6e8245c already in progress: creating tablet
I20240506 19:56:42.290316 13246 tablet_replica.cc:330] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:42.290553 13246 raft_consensus.cc:2229] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:42.290796 13246 pending_rounds.cc:62] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:42.290884 13246 pending_rounds.cc:68] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744046829481984 op_type: NO_OP noop_request { }
I20240506 19:56:42.291020 13246 raft_consensus.cc:2866] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:42.291272 13246 raft_consensus.cc:2258] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.292621 13245 tablet_bootstrap.cc:492] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:42.292788 13245 ts_tablet_manager.cc:1367] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.272s user 0.007s sys 0.000s
I20240506 19:56:42.293707 13245 raft_consensus.cc:348] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.293931 13245 raft_consensus.cc:374] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.294010 13245 raft_consensus.cc:729] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.294247 13245 consensus_queue.cc:260] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.295380 13245 ts_tablet_manager.cc:1398] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:42.295735 13245 tablet_bootstrap.cc:492] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:42.293471 13353 raft_consensus.cc:504] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.297765 13243 raft_consensus.cc:2258] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.298380 13243 ts_tablet_manager.cc:1875] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.299028 13353 leader_election.cc:288] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.300218 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:42.300550 13122 raft_consensus.cc:3037] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:42.301031 13791 tablet.cc:1722] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:42.304463 13245 tablet_bootstrap.cc:654] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.306210 13246 ts_tablet_manager.cc:1875] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.311957 13243 ts_tablet_manager.cc:1888] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.313619 13243 log.cc:1196] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ee59a76d52fc48d7bcf3256c359d3c02
I20240506 19:56:42.314254 13243 ts_tablet_manager.cc:1909] T ee59a76d52fc48d7bcf3256c359d3c02 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.315837 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ee59a76d52fc48d7bcf3256c359d3c02 (table default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7 [id=d9b546ebb0c146a7920cc05c08a29e32]) successfully deleted
I20240506 19:56:42.316802 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9e78a3376c14485f9a70307f926edaa5" } new_table_name: "default.Table_72ce5aee893c4fd78aca451ac2b7b773" modify_external_catalogs: true
W20240506 19:56:42.317523 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ff3049a9fa8049e5b84230e5e6e8245c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3049a9fa8049e5b84230e5e6e8245c already in progress: creating tablet
W20240506 19:56:42.318414 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9e3d13322b4a4a2db3b6d425679f68ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e3d13322b4a4a2db3b6d425679f68ed already in progress: creating tablet
I20240506 19:56:42.318588 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_05d52f607ff842c583d45eafbdcbf2ae" } new_table_name: "default.Table_322cb621e5734d8bb9694e30ac509ea7" modify_external_catalogs: true
W20240506 19:56:42.319285 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet de26793c006440dc9aecc07963ddb85e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de26793c006440dc9aecc07963ddb85e already in progress: creating tablet
I20240506 19:56:42.319783 13243 tablet_replica.cc:330] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.320009 13243 raft_consensus.cc:2229] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.320187 13243 raft_consensus.cc:2258] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.321718 13243 ts_tablet_manager.cc:1875] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.327105 13122 raft_consensus.cc:2454] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:42.328786 12922 leader_election.cc:302] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:42.329411 13353 raft_consensus.cc:2781] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:42.329972 13353 raft_consensus.cc:686] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:42.330715 13353 consensus_queue.cc:237] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.333468 13243 ts_tablet_manager.cc:1888] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.333607 13243 log.cc:1196] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f3d57249f30a44dab3384092ec20b515
I20240506 19:56:42.333962 13243 ts_tablet_manager.cc:1909] T f3d57249f30a44dab3384092ec20b515 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.334921 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f3d57249f30a44dab3384092ec20b515 (table default.Table_11b5bac0d01243f4a527f32ba62d5a38 [id=7197d7d0733346f4bc417c7dca54b79f]) successfully deleted
W20240506 19:56:42.338899 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bc85233b180f4b30a86bb65c86af684e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc85233b180f4b30a86bb65c86af684e already in progress: creating tablet
W20240506 19:56:42.339857 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f3a8531a1793404699532295164a3895 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a8531a1793404699532295164a3895 already in progress: creating tablet
W20240506 19:56:42.340838 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f28c1f55fd14aca86cb00e43b3a6b76 already in progress: creating tablet
I20240506 19:56:42.341461 13243 tablet_replica.cc:330] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.342056 13243 raft_consensus.cc:2229] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.342562 13243 raft_consensus.cc:2258] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.343327 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7714de27e50a4613bb9ad3a6a31b96aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.343607 13243 ts_tablet_manager.cc:1875] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:42.344177 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7714de27e50a4613bb9ad3a6a31b96aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.345288 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d5725b47e43244f0af881b07e145b3aa
I20240506 19:56:42.346424 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet d5725b47e43244f0af881b07e145b3aa with delete_type TABLET_DATA_DELETED (replaced by tablet 35453f75a3494f8baf13e0d5cacc9003) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.346860 12521 catalog_manager.cc:6554] ReplaceTablet: tablet d5725b47e43244f0af881b07e145b3aa deleted and replaced by tablet 35453f75a3494f8baf13e0d5cacc9003
I20240506 19:56:42.347302 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1d1d1064266d4b68934531d7c38d2801" } new_table_name: "default.Table_ab69ac48aa7a4934a9f383276277cf10" modify_external_catalogs: true
I20240506 19:56:42.357440 13243 ts_tablet_manager.cc:1888] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:42.357592 13243 log.cc:1196] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2d9230ed2e9a49ecb0a5e5d520cd03aa
I20240506 19:56:42.358003 13243 ts_tablet_manager.cc:1909] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.213249 13242 ts_tablet_manager.cc:1888] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:42.359154 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_bcd38427a79c48d1badd4f2536ff238f [id=f23583e607a4419986d06a38c45bbdf5]) successfully deleted
I20240506 19:56:42.361641 13243 tablet_replica.cc:330] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.362112 13243 raft_consensus.cc:2229] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.362576 13243 raft_consensus.cc:2258] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.363350 13243 ts_tablet_manager.cc:1875] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.364343 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet d5725b47e43244f0af881b07e145b3aa with delete_type TABLET_DATA_DELETED (replaced by tablet 35453f75a3494f8baf13e0d5cacc9003) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.372328 13243 ts_tablet_manager.cc:1888] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240506 19:56:42.372485 13243 log.cc:1196] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d5725b47e43244f0af881b07e145b3aa
I20240506 19:56:42.372875 13243 ts_tablet_manager.cc:1909] T d5725b47e43244f0af881b07e145b3aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.373250 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:42.378396 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 01c32eba2031446ab419cac64068068e (DEFAULT_TABLE table=default.Table_857904c3c3954ef5a113de49478bd27f [id=eb8c03717dc1454e8a944e84f309c31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.378944 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c32eba2031446ab419cac64068068e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.389066 13245 tablet_bootstrap.cc:492] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:42.389775 13245 ts_tablet_manager.cc:1367] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.094s user 0.008s sys 0.000s
I20240506 19:56:42.390167 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]) successfully deleted
I20240506 19:56:42.390916 13245 raft_consensus.cc:348] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.391249 13245 raft_consensus.cc:374] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.391978 13245 raft_consensus.cc:729] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.392362 13245 consensus_queue.cc:260] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.393218 13245 ts_tablet_manager.cc:1398] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20240506 19:56:42.401038 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 116df9bd6ab04a30ba09861e765d7267 consensus state has no leader
I20240506 19:56:42.223906 13716 consensus_queue.cc:237] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.286392 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:42.299340 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:42.406469 12921 leader_election.cc:341] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.265036 13244 tablet_bootstrap.cc:492] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:42.213447 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:42.407408 12921 leader_election.cc:341] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:42.233356 12783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 180230us
I20240506 19:56:42.410651 13244 ts_tablet_manager.cc:1367] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.450s user 0.001s sys 0.009s
I20240506 19:56:42.411564 13244 raft_consensus.cc:348] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.411792 13244 raft_consensus.cc:374] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.411895 13244 raft_consensus.cc:729] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.412492 13244 consensus_queue.cc:260] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.417479 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.418154 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet d5725b47e43244f0af881b07e145b3aa with delete_type TABLET_DATA_DELETED (replaced by tablet 35453f75a3494f8baf13e0d5cacc9003) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.418900 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 01c32eba2031446ab419cac64068068e (DEFAULT_TABLE table=default.Table_857904c3c3954ef5a113de49478bd27f [id=eb8c03717dc1454e8a944e84f309c31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.419328 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c32eba2031446ab419cac64068068e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.428202 13244 ts_tablet_manager.cc:1398] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240506 19:56:42.428546 13244 tablet_bootstrap.cc:492] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:42.430797 13244 tablet_bootstrap.cc:654] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.434026 13242 log.cc:1196] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1b7f56fea34946e590dae0621b28f9c6
I20240506 19:56:42.434751 13242 ts_tablet_manager.cc:1909] T 1b7f56fea34946e590dae0621b28f9c6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:42.435971 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 01c32eba2031446ab419cac64068068e (DEFAULT_TABLE table=default.Table_857904c3c3954ef5a113de49478bd27f [id=eb8c03717dc1454e8a944e84f309c31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.436439 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c32eba2031446ab419cac64068068e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.445936 13246 ts_tablet_manager.cc:1888] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:42.447105 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 301) took 4706 ms. Trace:
I20240506 19:56:42.457623 13249 tablet_bootstrap.cc:492] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.457904 13249 ts_tablet_manager.cc:1367] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.208s user 0.035s sys 0.082s
I20240506 19:56:42.458765 13249 raft_consensus.cc:348] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.459095 13249 raft_consensus.cc:374] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.459223 13249 raft_consensus.cc:729] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.459507 13249 consensus_queue.cc:260] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.460136 13249 ts_tablet_manager.cc:1398] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:42.460577 13249 tablet_bootstrap.cc:492] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.462316 13249 tablet_bootstrap.cc:654] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:42.463480 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 304a1401627948bbaff78fae4d42ab22 consensus state has no leader
I20240506 19:56:42.469710 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 08535fa1600b41ff919fefa93cfc4c78 (DEFAULT_TABLE table=default.Table_071b75a659614286996e871c603f9dcc [id=58c73d45cef84394a1dcabdd2716016f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.470366 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08535fa1600b41ff919fefa93cfc4c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.467901 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 08535fa1600b41ff919fefa93cfc4c78 (DEFAULT_TABLE table=default.Table_071b75a659614286996e871c603f9dcc [id=58c73d45cef84394a1dcabdd2716016f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.473942 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08535fa1600b41ff919fefa93cfc4c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.481415 13242 rpcz_store.cc:276] 0506 19:56:37.740244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.747612 (+ 7368us) service_pool.cc:225] Handling call
0506 19:56:41.919192 (+4171580us) ts_tablet_manager.cc:1120] Deleting tablet 1b7f56fea34946e590dae0621b28f9c6
0506 19:56:41.919202 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.210042 (+290840us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.213122 (+ 3080us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.447043 (+233921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4170598}
I20240506 19:56:42.482224 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1b7f56fea34946e590dae0621b28f9c6 (table default.Table_a2fb551b2fc541e38f0a655c4539060b [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:42.482836 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 307) took 4723 ms. Trace:
I20240506 19:56:42.483028 13242 rpcz_store.cc:276] 0506 19:56:37.758893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.758944 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:42.482678 (+4723734us) ts_tablet_manager.cc:1120] Deleting tablet 0ba2365d00044c32a06f9017a207adf7
0506 19:56:42.482688 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.482793 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4723012}
W20240506 19:56:42.483633 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0ba2365d00044c32a06f9017a207adf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ba2365d00044c32a06f9017a207adf7 already in progress: creating tablet
I20240506 19:56:42.484128 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 305) took 4725 ms. Trace:
I20240506 19:56:42.484287 13242 rpcz_store.cc:276] 0506 19:56:37.758591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.760913 (+ 2322us) service_pool.cc:225] Handling call
0506 19:56:42.483972 (+4723059us) ts_tablet_manager.cc:1120] Deleting tablet bafd3c4706f74e7da36a8f3538760f43
0506 19:56:42.483981 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.484092 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4722769}
W20240506 19:56:42.484733 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bafd3c4706f74e7da36a8f3538760f43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bafd3c4706f74e7da36a8f3538760f43 already in progress: creating tablet
I20240506 19:56:42.485384 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 308) took 4696 ms. Trace:
I20240506 19:56:42.485538 13242 rpcz_store.cc:276] 0506 19:56:37.788829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.788899 (+ 70us) service_pool.cc:225] Handling call
0506 19:56:42.485197 (+4696298us) ts_tablet_manager.cc:1120] Deleting tablet bd2c81ac92d04ddc93cc52e5c01cbc60
0506 19:56:42.485206 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.485322 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4685298}
W20240506 19:56:42.485976 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd2c81ac92d04ddc93cc52e5c01cbc60 already in progress: creating tablet
I20240506 19:56:42.486395 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 306) took 4727 ms. Trace:
I20240506 19:56:42.486550 13242 rpcz_store.cc:276] 0506 19:56:37.758754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.776073 (+ 17319us) service_pool.cc:225] Handling call
0506 19:56:42.486246 (+4710173us) ts_tablet_manager.cc:1120] Deleting tablet 0114edac976844619e438c17bb959010
0506 19:56:42.486256 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.486359 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4679320}
W20240506 19:56:42.487107 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0114edac976844619e438c17bb959010 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0114edac976844619e438c17bb959010 already in progress: creating tablet
I20240506 19:56:42.489641 13249 tablet_bootstrap.cc:492] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.448413 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 08535fa1600b41ff919fefa93cfc4c78 (DEFAULT_TABLE table=default.Table_071b75a659614286996e871c603f9dcc [id=58c73d45cef84394a1dcabdd2716016f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.489930 13249 ts_tablet_manager.cc:1367] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.016s
I20240506 19:56:42.490389 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08535fa1600b41ff919fefa93cfc4c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.491542 13249 raft_consensus.cc:348] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.493834 13249 raft_consensus.cc:374] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.493989 13249 raft_consensus.cc:729] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.494280 13249 consensus_queue.cc:260] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.494896 13249 ts_tablet_manager.cc:1398] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:42.495244 13249 tablet_bootstrap.cc:492] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.496867 13249 tablet_bootstrap.cc:654] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.456233 13245 tablet_bootstrap.cc:492] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:42.488472 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 1b89fb24d94042e1bfa772265a991680 (DEFAULT_TABLE table=default.Table_4180588e9c5d43cd95909d0f1cb7a140 [id=a9f6f784886543d7865427d6dbea071f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.498737 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b89fb24d94042e1bfa772265a991680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.508428 13245 tablet_bootstrap.cc:654] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.509577 13246 log.cc:1196] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ee59a76d52fc48d7bcf3256c359d3c02
I20240506 19:56:42.510074 13246 ts_tablet_manager.cc:1909] T ee59a76d52fc48d7bcf3256c359d3c02 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:42.511258 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ee59a76d52fc48d7bcf3256c359d3c02 (table default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7 [id=d9b546ebb0c146a7920cc05c08a29e32]) successfully deleted
W20240506 19:56:42.519021 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63f56d2180b1472a8168fb3a63589065 consensus state has no leader
I20240506 19:56:42.523444 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 1b89fb24d94042e1bfa772265a991680 (DEFAULT_TABLE table=default.Table_4180588e9c5d43cd95909d0f1cb7a140 [id=a9f6f784886543d7865427d6dbea071f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.524130 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 1b89fb24d94042e1bfa772265a991680 (DEFAULT_TABLE table=default.Table_4180588e9c5d43cd95909d0f1cb7a140 [id=a9f6f784886543d7865427d6dbea071f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.524531 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b89fb24d94042e1bfa772265a991680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.524533 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b89fb24d94042e1bfa772265a991680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.524777 13093 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e073296ce46e786362aaed96eb4dd (DEFAULT_TABLE table=default.Table_5f73ddfe064b4a9ea98eaa48ce44334f [id=792ffee112da484d9e58c584b911c67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.525168 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e073296ce46e786362aaed96eb4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.532919 12964 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e073296ce46e786362aaed96eb4dd (DEFAULT_TABLE table=default.Table_5f73ddfe064b4a9ea98eaa48ce44334f [id=792ffee112da484d9e58c584b911c67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.532914 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.533511 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.533537 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e073296ce46e786362aaed96eb4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.536423 12815 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e073296ce46e786362aaed96eb4dd (DEFAULT_TABLE table=default.Table_5f73ddfe064b4a9ea98eaa48ce44334f [id=792ffee112da484d9e58c584b911c67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.536926 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e073296ce46e786362aaed96eb4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.538276 13098 tablet_service.cc:1467] Processing CreateTablet for tablet ffbb87878d06417cbabf3c218728dbdc (DEFAULT_TABLE table=default.Table_54bc24e04c80487d9a93fd01ad679d73 [id=be9182dde02a4dab9010593c3ae1ad8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.538717 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbb87878d06417cbabf3c218728dbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.540321 12963 tablet_service.cc:1467] Processing CreateTablet for tablet ffbb87878d06417cbabf3c218728dbdc (DEFAULT_TABLE table=default.Table_54bc24e04c80487d9a93fd01ad679d73 [id=be9182dde02a4dab9010593c3ae1ad8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.540551 12826 tablet_service.cc:1467] Processing CreateTablet for tablet ffbb87878d06417cbabf3c218728dbdc (DEFAULT_TABLE table=default.Table_54bc24e04c80487d9a93fd01ad679d73 [id=be9182dde02a4dab9010593c3ae1ad8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.540717 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbb87878d06417cbabf3c218728dbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.541065 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbb87878d06417cbabf3c218728dbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.544443 13246 tablet_replica.cc:330] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:42.544735 13246 raft_consensus.cc:2229] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.545015 13246 raft_consensus.cc:2258] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.545668 13246 ts_tablet_manager.cc:1875] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:42.546094 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b6fa6add23b8403ba5a0a65ac6752a5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fa6add23b8403ba5a0a65ac6752a5b already in progress: creating tablet
I20240506 19:56:42.549692 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_001c9250e65b432f8e9d4c1a38bda161"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.550318 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_001c9250e65b432f8e9d4c1a38bda161 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.554093 13244 tablet_bootstrap.cc:492] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:42.554337 13244 ts_tablet_manager.cc:1367] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.005s
I20240506 19:56:42.555217 13244 raft_consensus.cc:348] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.555821 13244 raft_consensus.cc:374] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.555850 13095 tablet_service.cc:1467] Processing CreateTablet for tablet d0aae034f52645ba96827923f4fa305c (DEFAULT_TABLE table=default.Table_f72793ccf9b146a0baad8e0836ca260c [id=9d6272e3834b49648147d5d86f7ba1a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.555953 13244 raft_consensus.cc:729] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.556226 13244 consensus_queue.cc:260] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.557734 12963 tablet_service.cc:1467] Processing CreateTablet for tablet d0aae034f52645ba96827923f4fa305c (DEFAULT_TABLE table=default.Table_f72793ccf9b146a0baad8e0836ca260c [id=9d6272e3834b49648147d5d86f7ba1a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.558046 13244 ts_tablet_manager.cc:1398] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:42.558151 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aae034f52645ba96827923f4fa305c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.558364 13244 tablet_bootstrap.cc:492] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:42.560004 13244 tablet_bootstrap.cc:654] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.560410 12826 tablet_service.cc:1467] Processing CreateTablet for tablet d0aae034f52645ba96827923f4fa305c (DEFAULT_TABLE table=default.Table_f72793ccf9b146a0baad8e0836ca260c [id=9d6272e3834b49648147d5d86f7ba1a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.560853 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aae034f52645ba96827923f4fa305c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.561774 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4dcc036a20824736b16815bcb47645c4
I20240506 19:56:42.562551 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dcc036a20824736b16815bcb47645c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.563032 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dcc036a20824736b16815bcb47645c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.563421 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dcc036a20824736b16815bcb47645c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:42.563614 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4dcc036a20824736b16815bcb47645c4 because the tablet was not found. No further retry: Not found: Tablet not found: 4dcc036a20824736b16815bcb47645c4
I20240506 19:56:42.566128 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 315) took 4628 ms. Trace:
I20240506 19:56:42.566304 13242 rpcz_store.cc:276] 0506 19:56:37.937427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.937554 (+ 127us) service_pool.cc:225] Handling call
0506 19:56:42.565962 (+4628408us) ts_tablet_manager.cc:1120] Deleting tablet 1372be71e1d141ed89e58cdb806431a4
0506 19:56:42.565971 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.566075 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4620121}
I20240506 19:56:42.566620 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7fd4c95acc7243f38ef31d5d772d6cb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.566843 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1372be71e1d141ed89e58cdb806431a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1372be71e1d141ed89e58cdb806431a4 already in progress: creating tablet
I20240506 19:56:42.566789 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 314) took 4631 ms. Trace:
I20240506 19:56:42.566983 13242 rpcz_store.cc:276] 0506 19:56:37.935310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.935428 (+ 118us) service_pool.cc:225] Handling call
0506 19:56:42.566641 (+4631213us) ts_tablet_manager.cc:1120] Deleting tablet 00de09ddcdd948a6a3500a6c9d814c14
0506 19:56:42.566651 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.566759 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4610103}
W20240506 19:56:42.567216 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fd4c95acc7243f38ef31d5d772d6cb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:42.567503 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 00de09ddcdd948a6a3500a6c9d814c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00de09ddcdd948a6a3500a6c9d814c14 already in progress: creating tablet
I20240506 19:56:42.567574 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 316) took 4623 ms. Trace:
I20240506 19:56:42.567687 13242 rpcz_store.cc:276] 0506 19:56:37.944388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:37.944432 (+ 44us) service_pool.cc:225] Handling call
0506 19:56:42.567459 (+4623027us) ts_tablet_manager.cc:1120] Deleting tablet 6b59ad067c274fddbba5f71e89480dec
0506 19:56:42.567469 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.567545 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4608363}
I20240506 19:56:42.568049 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 330) took 4477 ms. Trace:
W20240506 19:56:42.568115 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 6b59ad067c274fddbba5f71e89480dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b59ad067c274fddbba5f71e89480dec already in progress: creating tablet
I20240506 19:56:42.568148 13242 rpcz_store.cc:276] 0506 19:56:38.090656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.343000 (+252344us) service_pool.cc:225] Handling call
0506 19:56:42.567938 (+4224938us) ts_tablet_manager.cc:1120] Deleting tablet f3e7329b22414da886dee4e1ec6bf3ad
0506 19:56:42.567946 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.568019 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4216530}
I20240506 19:56:42.568953 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 331) took 4177 ms. Trace:
I20240506 19:56:42.569329 13242 rpcz_store.cc:276] 0506 19:56:38.391366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.391451 (+ 85us) service_pool.cc:225] Handling call
0506 19:56:42.568563 (+4177112us) ts_tablet_manager.cc:1120] Deleting tablet ceeae70e0d1144ea9e3e0f8727c15e6b
0506 19:56:42.568588 (+ 25us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.568860 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4144386}
W20240506 19:56:42.569541 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f3e7329b22414da886dee4e1ec6bf3ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3e7329b22414da886dee4e1ec6bf3ad already in progress: creating tablet
I20240506 19:56:42.570284 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 340) took 3759 ms. Trace:
W20240506 19:56:42.570525 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ceeae70e0d1144ea9e3e0f8727c15e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceeae70e0d1144ea9e3e0f8727c15e6b already in progress: creating tablet
I20240506 19:56:42.572885 13245 tablet_bootstrap.cc:492] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:42.573766 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 7a396afff3b243ef84cb3dc3836fcdfd (DEFAULT_TABLE table=default.Table_d01da29ce66546ab88b2eba38a69c4d1 [id=9d511b62ea4b4e66a721a3508f15540e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.574297 13245 ts_tablet_manager.cc:1367] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.000s
I20240506 19:56:42.574975 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 7a396afff3b243ef84cb3dc3836fcdfd (DEFAULT_TABLE table=default.Table_d01da29ce66546ab88b2eba38a69c4d1 [id=9d511b62ea4b4e66a721a3508f15540e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.575383 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a396afff3b243ef84cb3dc3836fcdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.575363 13245 raft_consensus.cc:348] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.575595 13245 raft_consensus.cc:374] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.575678 13245 raft_consensus.cc:729] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.580264 13093 tablet_service.cc:1467] Processing CreateTablet for tablet a03f574aec984b83ab596e92a6f323fc (DEFAULT_TABLE table=default.Table_d9668767a7184407ac5f03e85864411e [id=e3a4e004e5c64784afdcc92d35a67b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.581609 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 1e22e5a015f74526b49363bc789cb4ed (DEFAULT_TABLE table=default.Table_04311b0062d745e3ae935a1abe99343e [id=fe54314cddeb4a78bc0d6dcf9ee04d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.582137 12957 tablet_service.cc:1467] Processing CreateTablet for tablet a03f574aec984b83ab596e92a6f323fc (DEFAULT_TABLE table=default.Table_d9668767a7184407ac5f03e85864411e [id=e3a4e004e5c64784afdcc92d35a67b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.582379 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bff7050bbc2e47cf823d25ec10b89718
I20240506 19:56:42.582937 12517 catalog_manager.cc:6554] ReplaceTablet: tablet bff7050bbc2e47cf823d25ec10b89718 deleted and replaced by tablet cc22261ab2034e818834ba97c2760dad
I20240506 19:56:42.582993 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03f574aec984b83ab596e92a6f323fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.583330 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7604b11105d14a889c9b95c21133e402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.583997 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7604b11105d14a889c9b95c21133e402 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.586325 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet bff7050bbc2e47cf823d25ec10b89718 with delete_type TABLET_DATA_DELETED (replaced by tablet cc22261ab2034e818834ba97c2760dad) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.586331 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet bff7050bbc2e47cf823d25ec10b89718 with delete_type TABLET_DATA_DELETED (replaced by tablet cc22261ab2034e818834ba97c2760dad) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:42.587608 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bff7050bbc2e47cf823d25ec10b89718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bff7050bbc2e47cf823d25ec10b89718 already in progress: creating tablet
I20240506 19:56:42.591799 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 1e22e5a015f74526b49363bc789cb4ed (DEFAULT_TABLE table=default.Table_04311b0062d745e3ae935a1abe99343e [id=fe54314cddeb4a78bc0d6dcf9ee04d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.592185 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e22e5a015f74526b49363bc789cb4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.592433 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 4dcc036a20824736b16815bcb47645c4 (DEFAULT_TABLE table=default.Table_0ea05891ea5e48a5b2a179a245749dc7 [id=92c4ae56a3f94df891e3ec92e2571a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.593668 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aae034f52645ba96827923f4fa305c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.593916 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e22e5a015f74526b49363bc789cb4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.594173 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a396afff3b243ef84cb3dc3836fcdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.594655 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dcc036a20824736b16815bcb47645c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.594739 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 4dcc036a20824736b16815bcb47645c4 (DEFAULT_TABLE table=default.Table_0ea05891ea5e48a5b2a179a245749dc7 [id=92c4ae56a3f94df891e3ec92e2571a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.595103 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dcc036a20824736b16815bcb47645c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.577284 12811 tablet_service.cc:1467] Processing CreateTablet for tablet a03f574aec984b83ab596e92a6f323fc (DEFAULT_TABLE table=default.Table_d9668767a7184407ac5f03e85864411e [id=e3a4e004e5c64784afdcc92d35a67b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.577462 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 7a396afff3b243ef84cb3dc3836fcdfd (DEFAULT_TABLE table=default.Table_d01da29ce66546ab88b2eba38a69c4d1 [id=9d511b62ea4b4e66a721a3508f15540e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.583295 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet bff7050bbc2e47cf823d25ec10b89718 with delete_type TABLET_DATA_DELETED (replaced by tablet cc22261ab2034e818834ba97c2760dad) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.595822 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03f574aec984b83ab596e92a6f323fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.596207 12964 tablet_service.cc:1467] Processing CreateTablet for tablet f078559df48f4d23869ddcb4b3fca9fc (DEFAULT_TABLE table=default.Table_1d3c02a3c0e64178939ba1d042908501 [id=9df2fb4afc724271b10df7b97324cbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.596586 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f078559df48f4d23869ddcb4b3fca9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.597609 13104 tablet_service.cc:1467] Processing CreateTablet for tablet f078559df48f4d23869ddcb4b3fca9fc (DEFAULT_TABLE table=default.Table_1d3c02a3c0e64178939ba1d042908501 [id=9df2fb4afc724271b10df7b97324cbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.598696 12813 tablet_service.cc:1467] Processing CreateTablet for tablet f078559df48f4d23869ddcb4b3fca9fc (DEFAULT_TABLE table=default.Table_1d3c02a3c0e64178939ba1d042908501 [id=9df2fb4afc724271b10df7b97324cbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.599000 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a396afff3b243ef84cb3dc3836fcdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.599196 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f078559df48f4d23869ddcb4b3fca9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.586860 13716 raft_consensus.cc:482] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.599406 13716 raft_consensus.cc:504] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.601264 13716 leader_election.cc:288] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.575939 13245 consensus_queue.cc:260] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:42.602460 12781 leader_election.cc:341] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 5516e8aad4e8474692b5b6c40bc483c1
I20240506 19:56:42.590615 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 1e22e5a015f74526b49363bc789cb4ed (DEFAULT_TABLE table=default.Table_04311b0062d745e3ae935a1abe99343e [id=fe54314cddeb4a78bc0d6dcf9ee04d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.604053 13245 ts_tablet_manager.cc:1398] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240506 19:56:42.604501 13245 tablet_bootstrap.cc:492] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:42.606640 13245 tablet_bootstrap.cc:654] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:42.612586 12783 leader_election.cc:341] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 5516e8aad4e8474692b5b6c40bc483c1
I20240506 19:56:42.614593 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c8ece0bd3a3b47ef88a0b4910f0efae9
I20240506 19:56:42.615332 12783 leader_election.cc:302] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:42.615532 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cddf9ad090cc4ac690a9a68b8600791e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.616036 13716 raft_consensus.cc:2726] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:42.616098 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cddf9ad090cc4ac690a9a68b8600791e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.616334 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.616629 12810 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.617486 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet db13b58d17c94e0f82052f823915d9af
I20240506 19:56:42.618377 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9898f6715c8a4ed183046f6a9e930d11" } new_table_name: "default.Table_319ec87ca4c4468aa4a97b8034323333" modify_external_catalogs: true
I20240506 19:56:42.618887 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet db13b58d17c94e0f82052f823915d9af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.619113 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e22e5a015f74526b49363bc789cb4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.620283 12810 tablet_service.cc:1514] Processing DeleteTablet for tablet db13b58d17c94e0f82052f823915d9af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.620891 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.621157 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet db13b58d17c94e0f82052f823915d9af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:42.621994 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8ece0bd3a3b47ef88a0b4910f0efae9 already in progress: creating tablet
W20240506 19:56:42.622289 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet db13b58d17c94e0f82052f823915d9af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db13b58d17c94e0f82052f823915d9af already in progress: creating tablet
I20240506 19:56:42.617547 13242 rpcz_store.cc:276] 0506 19:56:38.811006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.835472 (+ 24466us) service_pool.cc:225] Handling call
0506 19:56:42.569934 (+3734462us) ts_tablet_manager.cc:1120] Deleting tablet 17da01620b8b41008421b977455e0c9a
0506 19:56:42.569951 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.570190 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3718936}
I20240506 19:56:42.630501 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 4dcc036a20824736b16815bcb47645c4 (DEFAULT_TABLE table=default.Table_0ea05891ea5e48a5b2a179a245749dc7 [id=92c4ae56a3f94df891e3ec92e2571a58]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:42.631778 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 17da01620b8b41008421b977455e0c9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17da01620b8b41008421b977455e0c9a already in progress: creating tablet
I20240506 19:56:42.632596 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ecc88443c04b42d19cc7fba339eeeaf5
I20240506 19:56:42.632972 12519 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4] alter complete (version 2)
I20240506 19:56:42.635898 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc88443c04b42d19cc7fba339eeeaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.636323 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet b529bb9362924caca399491e7d9e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.637588 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc88443c04b42d19cc7fba339eeeaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:42.639055 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ecc88443c04b42d19cc7fba339eeeaf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecc88443c04b42d19cc7fba339eeeaf5 already in progress: creating tablet
I20240506 19:56:42.639761 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.639964 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d57249f30a44dab3384092ec20b515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.639791 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91888e88234cbdbd00f48dc1574653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.640147 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f078559df48f4d23869ddcb4b3fca9fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:42.640875 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa because the tablet was not found. No further retry: Not found: Tablet not found: 2d9230ed2e9a49ecb0a5e5d520cd03aa
I20240506 19:56:42.640400 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:42.641971 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f3d57249f30a44dab3384092ec20b515 because the tablet was not found. No further retry: Not found: Tablet not found: f3d57249f30a44dab3384092ec20b515
I20240506 19:56:42.642279 13243 tablet_replica.cc:330] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.642467 13243 raft_consensus.cc:2229] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.642658 13243 raft_consensus.cc:2258] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.643185 13243 ts_tablet_manager.cc:1875] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.648934 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 20c1590e36b4471d8e72f4b07410ee0f from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.649282 13246 ts_tablet_manager.cc:1888] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.649466 13246 log.cc:1196] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:42.649822 13246 ts_tablet_manager.cc:1909] T ca394b7f53ce445e8778dd25d0cf25c7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:42.650105 13243 ts_tablet_manager.cc:1888] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.650334 13243 log.cc:1196] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/4f91888e88234cbdbd00f48dc1574653
I20240506 19:56:42.650753 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 447) took 1008 ms. Trace:
I20240506 19:56:42.650792 13243 ts_tablet_manager.cc:1909] T 4f91888e88234cbdbd00f48dc1574653 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.650864 13246 rpcz_store.cc:276] 0506 19:56:41.641992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.642293 (+ 301us) service_pool.cc:225] Handling call
0506 19:56:42.544375 (+902082us) ts_tablet_manager.cc:1120] Deleting tablet ca394b7f53ce445e8778dd25d0cf25c7
0506 19:56:42.544384 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.595154 (+ 50770us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.649167 (+ 54013us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.650695 (+ 1528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":894085}
I20240506 19:56:42.651392 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 443) took 1015 ms. Trace:
I20240506 19:56:42.651504 13246 rpcz_store.cc:276] 0506 19:56:41.636400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.636452 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:42.651226 (+1014774us) ts_tablet_manager.cc:1120] Deleting tablet 458314e4653748e48cf646c72373fc3e
0506 19:56:42.651235 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.651343 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000636}
I20240506 19:56:42.651782 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet d9443cdbaa3c40299b66d216673c4b1a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.652091 13246 tablet_replica.cc:330] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:42.652225 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 4f91888e88234cbdbd00f48dc1574653 (table default.Table_4b6f407e4a464024aed5a1fb84c49849 [id=c7b8c2f7b9f74051b8207e0e127d930d]) successfully deleted
I20240506 19:56:42.652300 13246 raft_consensus.cc:2229] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.652485 13246 raft_consensus.cc:2258] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.652626 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ca394b7f53ce445e8778dd25d0cf25c7 (table default.Table_ca61f3a9aefa4fba8fea32d717de3fb1 [id=bf75c039ad324e50a2ccb889439e2b4c]) successfully deleted
I20240506 19:56:42.652689 13243 tablet_replica.cc:330] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.652885 13243 raft_consensus.cc:2229] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:42.652941 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 458314e4653748e48cf646c72373fc3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 458314e4653748e48cf646c72373fc3e already in progress: creating tablet
I20240506 19:56:42.653013 13246 ts_tablet_manager.cc:1875] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.653054 13243 raft_consensus.cc:2258] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:42.653194 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9e3d13322b4a4a2db3b6d425679f68ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e3d13322b4a4a2db3b6d425679f68ed already in progress: creating tablet
W20240506 19:56:42.653445 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet de26793c006440dc9aecc07963ddb85e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de26793c006440dc9aecc07963ddb85e already in progress: creating tablet
I20240506 19:56:42.654695 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8fc0fb56e9d5488e95d9534f003454ec" } new_table_name: "default.Table_840b5efc8ccf4fef961df12c95e98177" modify_external_catalogs: true
I20240506 19:56:42.655238 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_227676cc58be480489ac03fb71b0c527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.656396 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_227676cc58be480489ac03fb71b0c527 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.653549 13243 ts_tablet_manager.cc:1875] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:42.657146 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc22261ab2034e818834ba97c2760dad has no consensus state
I20240506 19:56:42.657897 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_138cdbf4ae874f409ca3414d505a1217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.658030 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8" } modify_external_catalogs: true
I20240506 19:56:42.658435 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03f574aec984b83ab596e92a6f323fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:42.660501 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_138cdbf4ae874f409ca3414d505a1217 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.661113 13249 tablet_bootstrap.cc:492] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.661335 13249 ts_tablet_manager.cc:1367] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.166s user 0.017s sys 0.021s
I20240506 19:56:42.662279 13249 raft_consensus.cc:348] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.662571 13249 raft_consensus.cc:374] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.662719 13249 raft_consensus.cc:729] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.663085 13249 consensus_queue.cc:260] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.663707 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_109dfbf34cd845b3b626bc0a00a23e52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.667511 13057 consensus_peers.cc:485] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d9230ed2e9a49ecb0a5e5d520cd03aa. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:42.667822 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dcc036a20824736b16815bcb47645c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.668478 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 341) took 3819 ms. Trace:
I20240506 19:56:42.669907 13249 ts_tablet_manager.cc:1398] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:42.671022 13242 rpcz_store.cc:276] 0506 19:56:38.811108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.834970 (+ 23862us) service_pool.cc:225] Handling call
0506 19:56:42.622623 (+3787653us) ts_tablet_manager.cc:1120] Deleting tablet 2b5a46b91cb742a994496648fbf87aa4
0506 19:56:42.622634 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.630473 (+ 7839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3769319}
I20240506 19:56:42.671583 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 343) took 3825 ms. Trace:
I20240506 19:56:42.671723 13242 rpcz_store.cc:276] 0506 19:56:38.846175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.851014 (+ 4839us) service_pool.cc:225] Handling call
0506 19:56:42.671387 (+3820373us) ts_tablet_manager.cc:1120] Deleting tablet 607b3fc2c17b4793af9b84c19066ec5e
0506 19:56:42.671400 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.671543 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3816734}
I20240506 19:56:42.672122 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 342) took 3860 ms. Trace:
I20240506 19:56:42.672250 13242 rpcz_store.cc:276] 0506 19:56:38.811192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:38.835653 (+ 24461us) service_pool.cc:225] Handling call
0506 19:56:42.671972 (+3836319us) ts_tablet_manager.cc:1120] Deleting tablet 971db6ac24d54efca95695ed1c90d4ed
0506 19:56:42.671982 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.672087 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3811789}
I20240506 19:56:42.672617 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 352) took 3438 ms. Trace:
I20240506 19:56:42.672729 13242 rpcz_store.cc:276] 0506 19:56:39.234030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.406096 (+172066us) service_pool.cc:225] Handling call
0506 19:56:42.672492 (+3266396us) ts_tablet_manager.cc:1120] Deleting tablet 1615f6167aae4dc096b9f563ab5e6215
0506 19:56:42.672500 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.672588 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3022802}
W20240506 19:56:42.672930 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_109dfbf34cd845b3b626bc0a00a23e52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.673043 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 353) took 3429 ms. Trace:
I20240506 19:56:42.673141 13242 rpcz_store.cc:276] 0506 19:56:39.243712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.425902 (+182190us) service_pool.cc:225] Handling call
0506 19:56:42.672931 (+3247029us) ts_tablet_manager.cc:1120] Deleting tablet d7be9788df8e47d285bd932d078c6a8b
0506 19:56:42.672938 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.673014 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011533}
I20240506 19:56:42.673511 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 354) took 3429 ms. Trace:
I20240506 19:56:42.673607 13242 rpcz_store.cc:276] 0506 19:56:39.243846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.426620 (+182774us) service_pool.cc:225] Handling call
0506 19:56:42.673376 (+3246756us) ts_tablet_manager.cc:1120] Deleting tablet f8951bf4873e4382bcd42a52368fb376
0506 19:56:42.673384 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.673477 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007986}
I20240506 19:56:42.664180 12513 catalog_manager.cc:5484] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:42.673908 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 355) took 3429 ms. Trace:
I20240506 19:56:42.674014 13242 rpcz_store.cc:276] 0506 19:56:39.243950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.438723 (+194773us) service_pool.cc:225] Handling call
0506 19:56:42.673791 (+3235068us) ts_tablet_manager.cc:1120] Deleting tablet 85bc7cfd006f4d4ca95541dab6873d32
0506 19:56:42.673798 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.673879 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3000123}
I20240506 19:56:42.674113 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
I20240506 19:56:42.674335 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 377) took 2933 ms. Trace:
I20240506 19:56:42.674429 13242 rpcz_store.cc:276] 0506 19:56:39.740725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:39.845217 (+104492us) service_pool.cc:225] Handling call
0506 19:56:42.674226 (+2829009us) ts_tablet_manager.cc:1120] Deleting tablet 6d04a1c5ee7b4d529fa2f4da4945ff09
0506 19:56:42.674234 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.674303 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2401487}
I20240506 19:56:42.665094 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_85795c658c664939838d720f20e0477a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.674682 13242 tablet_replica.cc:330] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:42.674885 13242 raft_consensus.cc:2229] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:42.674940 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85795c658c664939838d720f20e0477a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.675104 13242 raft_consensus.cc:2258] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.675729 13242 ts_tablet_manager.cc:1875] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.666141 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8" } modify_external_catalogs: true
I20240506 19:56:42.667395 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 1372be71e1d141ed89e58cdb806431a4 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.677309 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5a18ae8689874543a57ab917b1efaff0" } new_table_name: "default.Table_826f8c31905e4a76b9bd194c717a5853" modify_external_catalogs: true
I20240506 19:56:42.671000 12519 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:42.679176 13249 tablet_bootstrap.cc:492] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.679446 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5ef3ebc12c1f47278ef201cf63e5e39d" } new_table_name: "default.Table_e050c5de47634fe69b12743c8b30d5e9" modify_external_catalogs: true
I20240506 19:56:42.680161 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc88443c04b42d19cc7fba339eeeaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:42.680783 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d43da1d12ce647dcbcd31263932b2f54 consensus state has no leader
I20240506 19:56:42.682696 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 5988b89cb52e4f858b52862138bed025 with delete_type TABLET_DATA_DELETED (replaced by tablet 4973e600038742a6bec0bb2d8f3551fe) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.684070 13249 tablet_bootstrap.cc:654] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.684578 13244 tablet_bootstrap.cc:492] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:42.684751 13244 ts_tablet_manager.cc:1367] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.001s
I20240506 19:56:42.686784 13245 tablet_bootstrap.cc:492] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:42.686933 13245 ts_tablet_manager.cc:1367] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20240506 19:56:42.687680 13245 raft_consensus.cc:348] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.687870 13245 raft_consensus.cc:374] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.687945 13245 raft_consensus.cc:729] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.693661 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:56:42.694236 12516 catalog_manager.cc:6554] ReplaceTablet: tablet d9443cdbaa3c40299b66d216673c4b1a deleted and replaced by tablet ca23d6f0a9e0445cbd4d0e708c8de2d3
I20240506 19:56:42.695291 13246 ts_tablet_manager.cc:1888] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.695436 13246 log.cc:1196] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f3d57249f30a44dab3384092ec20b515
I20240506 19:56:42.695801 13246 ts_tablet_manager.cc:1909] T f3d57249f30a44dab3384092ec20b515 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:42.696774 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet d9443cdbaa3c40299b66d216673c4b1a with delete_type TABLET_DATA_DELETED (replaced by tablet ca23d6f0a9e0445cbd4d0e708c8de2d3) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.696820 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f3d57249f30a44dab3384092ec20b515 (table default.Table_11b5bac0d01243f4a527f32ba62d5a38 [id=7197d7d0733346f4bc417c7dca54b79f]) successfully deleted
I20240506 19:56:42.697098 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 5516e8aad4e8474692b5b6c40bc483c1 with delete_type TABLET_DATA_DELETED (replaced by tablet da41ef03182445f1946584c2bf015139) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.697485 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b24fa4663244a88dac9ea99fe38f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.698823 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 378759ca310f453fba45d48879e2dd1a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:42.698709 13244 raft_consensus.cc:348] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.686870 13243 ts_tablet_manager.cc:1888] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.699721 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2365d00044c32a06f9017a207adf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.694977 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet d9443cdbaa3c40299b66d216673c4b1a with delete_type TABLET_DATA_DELETED (replaced by tablet ca23d6f0a9e0445cbd4d0e708c8de2d3) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.700032 13243 log.cc:1196] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d6c656e16c1045dea79bc23b64fdd5e8
W20240506 19:56:42.701673 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2b5a46b91cb742a994496648fbf87aa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b5a46b91cb742a994496648fbf87aa4 already in progress: creating tablet
W20240506 19:56:42.701905 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 607b3fc2c17b4793af9b84c19066ec5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607b3fc2c17b4793af9b84c19066ec5e already in progress: creating tablet
W20240506 19:56:42.702072 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 971db6ac24d54efca95695ed1c90d4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 971db6ac24d54efca95695ed1c90d4ed already in progress: creating tablet
W20240506 19:56:42.702263 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1615f6167aae4dc096b9f563ab5e6215 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1615f6167aae4dc096b9f563ab5e6215 already in progress: creating tablet
W20240506 19:56:42.702452 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d7be9788df8e47d285bd932d078c6a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7be9788df8e47d285bd932d078c6a8b already in progress: creating tablet
W20240506 19:56:42.702647 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f8951bf4873e4382bcd42a52368fb376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8951bf4873e4382bcd42a52368fb376 already in progress: creating tablet
W20240506 19:56:42.702822 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 85bc7cfd006f4d4ca95541dab6873d32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85bc7cfd006f4d4ca95541dab6873d32 already in progress: creating tablet
W20240506 19:56:42.703032 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 already in progress: creating tablet
I20240506 19:56:42.703645 13243 ts_tablet_manager.cc:1909] T d6c656e16c1045dea79bc23b64fdd5e8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.704123 13353 raft_consensus.cc:482] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.704349 13353 raft_consensus.cc:504] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.705075 13353 leader_election.cc:288] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.705704 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f1e4efbe8b40b199fcb3020569e1a1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:42.705947 13122 raft_consensus.cc:2454] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:42.706029 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d6c656e16c1045dea79bc23b64fdd5e8 (table default.Table_c905679471c7485fa59180f110b5ec1f [id=c4769761819944109eebf88cb916cb27]) successfully deleted
W20240506 19:56:42.706359 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d9443cdbaa3c40299b66d216673c4b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9443cdbaa3c40299b66d216673c4b1a already in progress: creating tablet
I20240506 19:56:42.706521 12922 leader_election.cc:302] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:42.706897 13353 raft_consensus.cc:2781] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:42.706986 13353 raft_consensus.cc:482] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:42.707077 13353 raft_consensus.cc:3037] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:42.708353 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bc85233b180f4b30a86bb65c86af684e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc85233b180f4b30a86bb65c86af684e already in progress: creating tablet
I20240506 19:56:42.708596 13246 tablet_replica.cc:330] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:42.708645 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f3a8531a1793404699532295164a3895 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a8531a1793404699532295164a3895 already in progress: creating tablet
I20240506 19:56:42.708818 13246 raft_consensus.cc:2229] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
W20240506 19:56:42.708904 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f28c1f55fd14aca86cb00e43b3a6b76 already in progress: creating tablet
I20240506 19:56:42.709209 13246 raft_consensus.cc:2258] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.709169 13716 raft_consensus.cc:482] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.709739 13716 raft_consensus.cc:504] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.710067 13353 raft_consensus.cc:504] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.710983 13244 raft_consensus.cc:374] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.711301 13246 ts_tablet_manager.cc:1875] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.711362 13244 raft_consensus.cc:729] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.711565 13353 leader_election.cc:288] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.711306 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f1e4efbe8b40b199fcb3020569e1a1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:42.711728 13122 raft_consensus.cc:3037] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:42.711640 13244 consensus_queue.cc:260] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.713701 13245 consensus_queue.cc:260] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.714335 13122 raft_consensus.cc:2454] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:42.714783 12922 leader_election.cc:302] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:42.715000 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f1e4efbe8b40b199fcb3020569e1a1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:42.715138 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d819d1237f4798b4847fdf378cbc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.715309 13353 raft_consensus.cc:2781] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:42.715451 13353 raft_consensus.cc:686] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:42.716159 13353 consensus_queue.cc:237] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:42.716573 12921 leader_election.cc:341] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.716874 13716 leader_election.cc:288] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.717032 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet d9443cdbaa3c40299b66d216673c4b1a with delete_type TABLET_DATA_DELETED (replaced by tablet ca23d6f0a9e0445cbd4d0e708c8de2d3) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.718737 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f1e4efbe8b40b199fcb3020569e1a1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:42.719121 13244 ts_tablet_manager.cc:1398] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
W20240506 19:56:42.718734 12781 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 27b24fa4663244a88dac9ea99fe38f3a
W20240506 19:56:42.719638 12921 leader_election.cc:341] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:42.724089 12783 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 27b24fa4663244a88dac9ea99fe38f3a
I20240506 19:56:42.724570 13245 ts_tablet_manager.cc:1398] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240506 19:56:42.724761 12783 leader_election.cc:302] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:42.725792 13249 tablet_bootstrap.cc:492] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.725958 13249 ts_tablet_manager.cc:1367] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20240506 19:56:42.730790 13249 raft_consensus.cc:348] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.731086 13249 raft_consensus.cc:374] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.731173 13249 raft_consensus.cc:729] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.731443 13249 consensus_queue.cc:260] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.731961 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_59add447dae84c509690e4f1f689fb40" } new_table_name: "default.Table_54d5d8d8437a44a4a66e103564486b0d" modify_external_catalogs: true
I20240506 19:56:42.733922 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 1667a0bbd3af4d678434b068c47cc518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.734210 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 1488328703024ae8983add9e4d6afc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.734647 13243 tablet_replica.cc:330] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.734838 13243 raft_consensus.cc:2229] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.735100 13243 raft_consensus.cc:2258] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.735265 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 0114edac976844619e438c17bb959010 with delete_type TABLET_DATA_DELETED (replaced by tablet 6352d2c630204df3a24e299e167f92d4) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.735951 13243 ts_tablet_manager.cc:1875] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.736569 13244 tablet_bootstrap.cc:492] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:42.739346 13716 raft_consensus.cc:2726] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.741559 13245 tablet_bootstrap.cc:492] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:42.742045 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3bed0eb234e21b890242e9d23a5b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.745450 13244 tablet_bootstrap.cc:654] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.748301 13353 raft_consensus.cc:482] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.748545 13353 raft_consensus.cc:504] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:42.750451 12922 leader_election.cc:341] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:42.750712 13353 leader_election.cc:288] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.752120 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca394b7f53ce445e8778dd25d0cf25c7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:42.752635 12921 leader_election.cc:341] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.752768 12921 leader_election.cc:302] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:42.755194 13242 ts_tablet_manager.cc:1888] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:42.755584 13353 raft_consensus.cc:2726] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.761250 13243 ts_tablet_manager.cc:1888] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:42.761394 13242 log.cc:1196] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/320e714ed6b64f2d84770e7182e4103a
I20240506 19:56:42.764499 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f6ec20e42cd94faebb5b13d490069e9d" } modify_external_catalogs: true
I20240506 19:56:42.764575 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_67154b63102e41e0b35410affe262e62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.772140 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f6ec20e42cd94faebb5b13d490069e9d" } modify_external_catalogs: true
I20240506 19:56:42.772660 13242 ts_tablet_manager.cc:1909] T 320e714ed6b64f2d84770e7182e4103a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:42.772670 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67154b63102e41e0b35410affe262e62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.767889 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9c18888ae3d6420ca1de3b13fc36cf6c
I20240506 19:56:42.773804 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 378) took 3024 ms. Trace:
I20240506 19:56:42.773911 13242 rpcz_store.cc:276] 0506 19:56:39.749839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.272771 (+522932us) service_pool.cc:225] Handling call
0506 19:56:42.674624 (+2401853us) ts_tablet_manager.cc:1120] Deleting tablet 320e714ed6b64f2d84770e7182e4103a
0506 19:56:42.674634 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.727500 (+ 52866us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.755055 (+ 27555us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:42.773717 (+ 18662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2401613}
I20240506 19:56:42.774538 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 379) took 3023 ms. Trace:
I20240506 19:56:42.774626 13242 rpcz_store.cc:276] 0506 19:56:39.751291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.273044 (+521753us) service_pool.cc:225] Handling call
0506 19:56:42.774343 (+2501299us) ts_tablet_manager.cc:1120] Deleting tablet 4095f9fc28ff48f68064ec8318ff7b51
0506 19:56:42.774352 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.774461 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2501118}
I20240506 19:56:42.775080 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 380) took 3023 ms. Trace:
I20240506 19:56:42.775049 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 320e714ed6b64f2d84770e7182e4103a (table default.Table_8be586b55bf2436698bfeb2f94fb465d [id=090dfa9462424be1b846e1cb05569156]) successfully deleted
I20240506 19:56:42.775161 13242 rpcz_store.cc:276] 0506 19:56:39.751404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.273240 (+521836us) service_pool.cc:225] Handling call
0506 19:56:42.774937 (+2501697us) ts_tablet_manager.cc:1120] Deleting tablet 997ecba7fbae41178a25bf7af4d152e1
0506 19:56:42.774945 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.775036 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2498336}
W20240506 19:56:42.775318 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4095f9fc28ff48f68064ec8318ff7b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4095f9fc28ff48f68064ec8318ff7b51 already in progress: creating tablet
I20240506 19:56:42.775441 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 385) took 2473 ms. Trace:
I20240506 19:56:42.775519 13242 rpcz_store.cc:276] 0506 19:56:40.302141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.451427 (+149286us) service_pool.cc:225] Handling call
0506 19:56:42.775328 (+2323901us) ts_tablet_manager.cc:1120] Deleting tablet 2519c03738864f6ab5dcfab7bc3fc8aa
0506 19:56:42.775335 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.775395 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323697}
I20240506 19:56:42.775753 13242 tablet_replica.cc:330] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:42.775935 13242 raft_consensus.cc:2229] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.776135 13242 raft_consensus.cc:2258] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.776697 13242 ts_tablet_manager.cc:1875] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:42.777609 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 997ecba7fbae41178a25bf7af4d152e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 997ecba7fbae41178a25bf7af4d152e1 already in progress: creating tablet
W20240506 19:56:42.777873 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2519c03738864f6ab5dcfab7bc3fc8aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2519c03738864f6ab5dcfab7bc3fc8aa already in progress: creating tablet
I20240506 19:56:42.778178 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c18888ae3d6420ca1de3b13fc36cf6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:42.770599 13245 tablet_bootstrap.cc:654] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:42.780812 13243 log.cc:1196] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:42.776080 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c18888ae3d6420ca1de3b13fc36cf6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.765347 13249 ts_tablet_manager.cc:1398] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20240506 19:56:42.779206 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c18888ae3d6420ca1de3b13fc36cf6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.783370 13249 tablet_bootstrap.cc:492] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.785018 13249 tablet_bootstrap.cc:654] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:42.786262 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 986f9c83b04f41b49e9b78fce731a90f consensus state has no leader
I20240506 19:56:42.786801 13353 raft_consensus.cc:482] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.787098 13353 raft_consensus.cc:504] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.788332 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1488328703024ae8983add9e4d6afc8f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:42.788924 12921 leader_election.cc:341] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:42.790069 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20aa06290b9a4814a006c423ccd8fc46 consensus state has no leader
W20240506 19:56:42.796824 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07c7c7b2da9e403f8241a512e210c4d9 consensus state has no leader
W20240506 19:56:42.802588 12922 leader_election.cc:341] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1488328703024ae8983add9e4d6afc8f
I20240506 19:56:42.802765 12922 leader_election.cc:302] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:42.803050 13629 raft_consensus.cc:2726] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.803380 13353 leader_election.cc:288] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:42.806970 13244 tablet_bootstrap.cc:492] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:42.806447 13243 ts_tablet_manager.cc:1909] T 1667a0bbd3af4d678434b068c47cc518 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:42.808632 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1667a0bbd3af4d678434b068c47cc518 (table default.Table_dfd40b6029db498e870d3e8a44ae6bdd [id=c4a20cff71d7496e8b07f477a0c1ea1d]) successfully deleted
I20240506 19:56:42.809852 13730 raft_consensus.cc:482] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:42.810034 13730 raft_consensus.cc:504] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.810729 13730 leader_election.cc:288] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:42.812014 13057 leader_election.cc:341] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 5988b89cb52e4f858b52862138bed025
I20240506 19:56:42.812363 13243 tablet_replica.cc:330] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:42.812615 13243 raft_consensus.cc:2229] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.813402 13244 ts_tablet_manager.cc:1367] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.003s
I20240506 19:56:42.813995 13243 raft_consensus.cc:2258] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.816735 13246 ts_tablet_manager.cc:1888] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:42.817349 12521 catalog_manager.cc:5484] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:42.817981 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:42.818073 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5988b89cb52e4f858b52862138bed025" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:42.819274 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_72ce5aee893c4fd78aca451ac2b7b773" } modify_external_catalogs: true
I20240506 19:56:42.819500 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_72ce5aee893c4fd78aca451ac2b7b773" } modify_external_catalogs: true
I20240506 19:56:42.820318 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 597a48f6950844299f5f292e6a101cae (DEFAULT_TABLE table=default.Table_4cc2d777e8ad43d49ca9092206e8dc72 [id=74dfbbbb284d48a8bb4fb2626eb67a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.820778 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 597a48f6950844299f5f292e6a101cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.821007 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 4be4e628b9364cccb525b8e1b8070286 (DEFAULT_TABLE table=default.Table_11b0e013ac1943e3ad3061b8a451c8d8 [id=7eb1f2d3378d4eb9b2d07ec9e46e5efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.821399 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be4e628b9364cccb525b8e1b8070286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.823318 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 597a48f6950844299f5f292e6a101cae (DEFAULT_TABLE table=default.Table_4cc2d777e8ad43d49ca9092206e8dc72 [id=74dfbbbb284d48a8bb4fb2626eb67a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.828078 12955 tablet_service.cc:1467] Processing CreateTablet for tablet c5f3def738fb4cb195d08e48d7d2b293 (DEFAULT_TABLE table=default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.828476 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f3def738fb4cb195d08e48d7d2b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.829633 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f48001b944b84606b9e9e95a00e8a6a7" } modify_external_catalogs: true
I20240506 19:56:42.829860 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f48001b944b84606b9e9e95a00e8a6a7" } modify_external_catalogs: true
I20240506 19:56:42.833240 12958 tablet_service.cc:1467] Processing CreateTablet for tablet b6eac98ac8f040c58f61106bb0e989de (DEFAULT_TABLE table=default.Table_4ac15d4e3f6b4719b8b728be5474dd63 [id=15c9ba78bdf345b9b8838a4aeaebb885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.833765 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd (DEFAULT_TABLE table=default.Table_730c6ad3babb49aaba5d81f48381daec [id=47243ad822944b879acaef4663ee6dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.834390 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 09cb71b5962f4561bf0fdbe12bf45296 (DEFAULT_TABLE table=default.Table_574f459d109b430ea8c288af9280052f [id=f356817895cf4ca58705c402fbb18f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.834822 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd12a9cc3924f3eb8aec502e9fa2d47 (DEFAULT_TABLE table=default.Table_ca9a4ec248934294830f719bc5d46204 [id=b39d38ddf7f4433fb3fb1ed1c6400a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.835395 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6eac98ac8f040c58f61106bb0e989de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.836387 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd12a9cc3924f3eb8aec502e9fa2d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.836462 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_227676cc58be480489ac03fb71b0c527" } modify_external_catalogs: true
I20240506 19:56:42.836663 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_227676cc58be480489ac03fb71b0c527" } modify_external_catalogs: true
I20240506 19:56:42.836905 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09cb71b5962f4561bf0fdbe12bf45296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.837332 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.837651 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.838246 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.839876 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1aacc41134ef4e40ac2c05d45c1faf3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.840379 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1aacc41134ef4e40ac2c05d45c1faf3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.817745 13243 ts_tablet_manager.cc:1875] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.848747 13249 tablet_bootstrap.cc:492] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:42.848929 13249 ts_tablet_manager.cc:1367] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20240506 19:56:42.849829 13249 raft_consensus.cc:348] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.850075 13249 raft_consensus.cc:374] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.850162 13249 raft_consensus.cc:729] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.850404 13249 consensus_queue.cc:260] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.851004 13249 ts_tablet_manager.cc:1398] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:42.851549 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 378759ca310f453fba45d48879e2dd1a
I20240506 19:56:42.852031 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 378759ca310f453fba45d48879e2dd1a deleted and replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f
I20240506 19:56:42.854604 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f249dc790e774c46a2cf5e2af4f2a7a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.855258 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f249dc790e774c46a2cf5e2af4f2a7a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.855202 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ea8710b001ee40d9aec404921fb4dcd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:42.855751 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea8710b001ee40d9aec404921fb4dcd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.857733 13249 tablet_bootstrap.cc:492] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:42.858917 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 35453f75a3494f8baf13e0d5cacc9003 (DEFAULT_TABLE table=default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.859375 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35453f75a3494f8baf13e0d5cacc9003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.859439 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 4216f98cb6ac4e249ca2653518ff9e87 (DEFAULT_TABLE table=default.Table_5553b476197648cd9965dd87bbabb42f [id=00e705e26d69454c980a847fe849a36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.859553 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_88a94bbd43f4495aac452adfdfa73c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:42.859771 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4216f98cb6ac4e249ca2653518ff9e87. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:42.860074 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88a94bbd43f4495aac452adfdfa73c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:42.862092 13246 log.cc:1196] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2d9230ed2e9a49ecb0a5e5d520cd03aa
W20240506 19:56:42.818784 13058 leader_election.cc:341] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:42.862594 13246 ts_tablet_manager.cc:1909] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:42.862530 13058 leader_election.cc:302] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:42.862912 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 44144.1us
I20240506 19:56:42.862885 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7fd4c95acc7243f38ef31d5d772d6cb5" } new_table_name: "default.Table_300911c21c1140aca1cd2170583017fb" modify_external_catalogs: true
I20240506 19:56:42.827126 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 4be4e628b9364cccb525b8e1b8070286 (DEFAULT_TABLE table=default.Table_11b0e013ac1943e3ad3061b8a451c8d8 [id=7eb1f2d3378d4eb9b2d07ec9e46e5efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.863354 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be4e628b9364cccb525b8e1b8070286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.863718 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_bcd38427a79c48d1badd4f2536ff238f [id=f23583e607a4419986d06a38c45bbdf5]) successfully deleted
I20240506 19:56:42.827512 13104 tablet_service.cc:1467] Processing CreateTablet for tablet c5f3def738fb4cb195d08e48d7d2b293 (DEFAULT_TABLE table=default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.827881 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 597a48f6950844299f5f292e6a101cae (DEFAULT_TABLE table=default.Table_4cc2d777e8ad43d49ca9092206e8dc72 [id=74dfbbbb284d48a8bb4fb2626eb67a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.828100 13087 tablet_service.cc:1467] Processing CreateTablet for tablet b6eac98ac8f040c58f61106bb0e989de (DEFAULT_TABLE table=default.Table_4ac15d4e3f6b4719b8b728be5474dd63 [id=15c9ba78bdf345b9b8838a4aeaebb885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.828819 13245 tablet_bootstrap.cc:492] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:42.865628 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f3def738fb4cb195d08e48d7d2b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.829990 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd (DEFAULT_TABLE table=default.Table_730c6ad3babb49aaba5d81f48381daec [id=47243ad822944b879acaef4663ee6dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.865840 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6eac98ac8f040c58f61106bb0e989de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.865849 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3291123a1e42af80dbfb578c855618 (DEFAULT_TABLE table=default.Table_012b93e45a4945ee964f19ccd6bbfa0d [id=c30426d9251f45ea93b91e907e358bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.866017 13730 raft_consensus.cc:2726] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:42.831045 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 09cb71b5962f4561bf0fdbe12bf45296 (DEFAULT_TABLE table=default.Table_574f459d109b430ea8c288af9280052f [id=f356817895cf4ca58705c402fbb18f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.866138 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.832417 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd12a9cc3924f3eb8aec502e9fa2d47 (DEFAULT_TABLE table=default.Table_ca9a4ec248934294830f719bc5d46204 [id=b39d38ddf7f4433fb3fb1ed1c6400a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.841325 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 35453f75a3494f8baf13e0d5cacc9003 (DEFAULT_TABLE table=default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.866320 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3291123a1e42af80dbfb578c855618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.841707 13086 tablet_service.cc:1467] Processing CreateTablet for tablet cc22261ab2034e818834ba97c2760dad (DEFAULT_TABLE table=default.Table_319ec87ca4c4468aa4a97b8034323333 [id=5faa78277f6641439177ede005a94e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.842018 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 58c88f80f5b34672b7c3a4b604469bf1 (DEFAULT_TABLE table=default.Table_7714de27e50a4613bb9ad3a6a31b96aa [id=d25fcde9ffaa4b64a0322b921aacc8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.834024 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 4216f98cb6ac4e249ca2653518ff9e87 (DEFAULT_TABLE table=default.Table_5553b476197648cd9965dd87bbabb42f [id=00e705e26d69454c980a847fe849a36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.842345 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 0b899244c35b4bbe9e916c778111f0c0 (DEFAULT_TABLE table=default.Table_dafad97b193f40038dfd516c93b79e96 [id=1a1d40c83b0744689cb8a85164109e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.865849 13245 ts_tablet_manager.cc:1367] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20240506 19:56:42.866667 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd12a9cc3924f3eb8aec502e9fa2d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.866897 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c88f80f5b34672b7c3a4b604469bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.869455 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09cb71b5962f4561bf0fdbe12bf45296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.869716 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4216f98cb6ac4e249ca2653518ff9e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.854192 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 378759ca310f453fba45d48879e2dd1a with delete_type TABLET_DATA_DELETED (replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.873517 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 597a48f6950844299f5f292e6a101cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.873932 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35453f75a3494f8baf13e0d5cacc9003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.886168 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b899244c35b4bbe9e916c778111f0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.889386 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 597a48f6950844299f5f292e6a101cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.890475 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 378759ca310f453fba45d48879e2dd1a with delete_type TABLET_DATA_DELETED (replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.891366 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 0114edac976844619e438c17bb959010 with delete_type TABLET_DATA_DELETED (replaced by tablet 6352d2c630204df3a24e299e167f92d4) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.891525 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 58c88f80f5b34672b7c3a4b604469bf1 (DEFAULT_TABLE table=default.Table_7714de27e50a4613bb9ad3a6a31b96aa [id=d25fcde9ffaa4b64a0322b921aacc8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.893587 12962 tablet_service.cc:1467] Processing CreateTablet for tablet cc22261ab2034e818834ba97c2760dad (DEFAULT_TABLE table=default.Table_319ec87ca4c4468aa4a97b8034323333 [id=5faa78277f6641439177ede005a94e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.894393 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.899837 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 0b899244c35b4bbe9e916c778111f0c0 (DEFAULT_TABLE table=default.Table_dafad97b193f40038dfd516c93b79e96 [id=1a1d40c83b0744689cb8a85164109e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.900233 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3291123a1e42af80dbfb578c855618 (DEFAULT_TABLE table=default.Table_012b93e45a4945ee964f19ccd6bbfa0d [id=c30426d9251f45ea93b91e907e358bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.900570 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c88f80f5b34672b7c3a4b604469bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.901461 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3291123a1e42af80dbfb578c855618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.902153 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b899244c35b4bbe9e916c778111f0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.902721 12948 tablet_service.cc:1467] Processing CreateTablet for tablet ee69d8e09c2f4c898ce547ec396ca0e6 (DEFAULT_TABLE table=default.Table_942245e6c2a94a96af9206be9765c407 [id=38df61b9f4a641eeb68e44b1eadf7d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.903848 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee69d8e09c2f4c898ce547ec396ca0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.905261 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc22261ab2034e818834ba97c2760dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.908712 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 116df9bd6ab04a30ba09861e765d7267
I20240506 19:56:42.911329 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b2887df9eaee49beba57959a096c0c65" } new_table_name: "default.Table_8fe0b52e5f1640da9f2920e67e7e26bb" modify_external_catalogs: true
I20240506 19:56:42.918004 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd (DEFAULT_TABLE table=default.Table_730c6ad3babb49aaba5d81f48381daec [id=47243ad822944b879acaef4663ee6dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.919476 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc22261ab2034e818834ba97c2760dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.920774 13097 tablet_service.cc:1467] Processing CreateTablet for tablet ee69d8e09c2f4c898ce547ec396ca0e6 (DEFAULT_TABLE table=default.Table_942245e6c2a94a96af9206be9765c407 [id=38df61b9f4a641eeb68e44b1eadf7d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.922045 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb8bb09098d401da62ad737ffb697ad (DEFAULT_TABLE table=default.Table_373183be2430417d8304f5e10cc72a57 [id=65fbfa6971874ba9a9f388961010594e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.922837 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb8bb09098d401da62ad737ffb697ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.926736 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa (DEFAULT_TABLE table=default.Table_4674aa2f59294605bc41516bbae12158 [id=ea63555353cd45d485b18c833792fda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.927876 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.928488 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb8bb09098d401da62ad737ffb697ad (DEFAULT_TABLE table=default.Table_373183be2430417d8304f5e10cc72a57 [id=65fbfa6971874ba9a9f388961010594e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.928638 13095 tablet_service.cc:1467] Processing CreateTablet for tablet e6d37bb1c2144c4fa71aab6113b9f06e (DEFAULT_TABLE table=default.Table_195d90ff34ff4ca6b6282483c9691f47 [id=99717b6265bc402089da1d146125f883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.928790 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 116df9bd6ab04a30ba09861e765d7267 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.929049 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa (DEFAULT_TABLE table=default.Table_4674aa2f59294605bc41516bbae12158 [id=ea63555353cd45d485b18c833792fda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.929440 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 116df9bd6ab04a30ba09861e765d7267 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:42.942121 13245 raft_consensus.cc:348] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.942554 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet ea25fb8e42234b82885b8fb2275d4925 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:42.954034 13246 tablet_replica.cc:330] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:42.966609 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee69d8e09c2f4c898ce547ec396ca0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.966778 12957 tablet_service.cc:1467] Processing CreateTablet for tablet e6d37bb1c2144c4fa71aab6113b9f06e (DEFAULT_TABLE table=default.Table_195d90ff34ff4ca6b6282483c9691f47 [id=99717b6265bc402089da1d146125f883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.967242 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d37bb1c2144c4fa71aab6113b9f06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.968197 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.968658 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb8bb09098d401da62ad737ffb697ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.969132 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d37bb1c2144c4fa71aab6113b9f06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:42.983465 13245 raft_consensus.cc:374] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:42.983700 13245 raft_consensus.cc:729] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:42.968134 13246 raft_consensus.cc:2229] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:42.984182 13246 raft_consensus.cc:2258] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:42.990373 13246 ts_tablet_manager.cc:1875] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:42.986501 13245 consensus_queue.cc:260] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:42.996966 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4b4ad903e75547fa829b82cc722fef1e
I20240506 19:56:42.996958 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 09cb71b5962f4561bf0fdbe12bf45296 (DEFAULT_TABLE table=default.Table_574f459d109b430ea8c288af9280052f [id=f356817895cf4ca58705c402fbb18f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:42.997435 13245 ts_tablet_manager.cc:1398] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.131s user 0.001s sys 0.002s
I20240506 19:56:42.998342 12519 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 55fdced0feb949a298bfd5fba77940d4
I20240506 19:56:43.000829 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4ad903e75547fa829b82cc722fef1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.004079 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 4be4e628b9364cccb525b8e1b8070286 (DEFAULT_TABLE table=default.Table_11b0e013ac1943e3ad3061b8a451c8d8 [id=7eb1f2d3378d4eb9b2d07ec9e46e5efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.006076 13245 tablet_bootstrap.cc:492] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:43.007431 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet ef03fcaa05d04ebd9f262f43ff7c02e4 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:43.007786 13245 tablet_bootstrap.cc:654] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.009078 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ee7192b3b36648858181f492bf1a72b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.009658 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee7192b3b36648858181f492bf1a72b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:43.010982 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ea8f5301f524e4e8910ce61837a1dd4 has no consensus state
I20240506 19:56:43.011377 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2b9776133219459d8dfd0ea02428d80f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.011898 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b9776133219459d8dfd0ea02428d80f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.012065 13243 ts_tablet_manager.cc:1888] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.012869 13243 log.cc:1196] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1488328703024ae8983add9e4d6afc8f
I20240506 19:56:43.013273 13243 ts_tablet_manager.cc:1909] T 1488328703024ae8983add9e4d6afc8f P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:43.014261 13243 tablet_replica.cc:330] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:43.014482 13243 raft_consensus.cc:2229] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:43.014848 13243 raft_consensus.cc:2258] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.015435 13243 ts_tablet_manager.cc:1875] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.015960 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_50e680b80e9e48faa1d0e0e07bf41693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.016520 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50e680b80e9e48faa1d0e0e07bf41693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.018112 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 64485c3c13e94055a019f2ef7be1f2fe
I20240506 19:56:43.018779 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_384bed2d6d984f9d88c62381e38b6321" } modify_external_catalogs: true
I20240506 19:56:43.018968 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_384bed2d6d984f9d88c62381e38b6321" } modify_external_catalogs: true
I20240506 19:56:43.020510 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6f7629baef0946e6aa92b1cbdef4d362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.021062 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f7629baef0946e6aa92b1cbdef4d362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.022683 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 64485c3c13e94055a019f2ef7be1f2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.023443 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4adca98d503a4754b9f11888dc549378" } new_table_name: "default.Table_dfb78f8b470b46f485886ae5bd7833f6" modify_external_catalogs: true
I20240506 19:56:43.024019 13246 ts_tablet_manager.cc:1888] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240506 19:56:43.024160 13246 log.cc:1196] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d5725b47e43244f0af881b07e145b3aa
I20240506 19:56:43.024610 13246 ts_tablet_manager.cc:1909] T d5725b47e43244f0af881b07e145b3aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:43.025228 13249 tablet_bootstrap.cc:654] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.026118 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_60d08e70820949809d66466222b3742d" } new_table_name: "default.Table_c9f790860bb44630a84fb411e08d50d5" modify_external_catalogs: true
I20240506 19:56:43.032194 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]) successfully deleted
W20240506 19:56:43.033874 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4dcc036a20824736b16815bcb47645c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dcc036a20824736b16815bcb47645c4 already in progress: creating tablet
W20240506 19:56:43.034791 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bff7050bbc2e47cf823d25ec10b89718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bff7050bbc2e47cf823d25ec10b89718 already in progress: creating tablet
I20240506 19:56:43.035187 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1488328703024ae8983add9e4d6afc8f (table default.Table_3501606fb8de42f68fd7894fa92e1a95 [id=3cabe6d9486d4b6fa610e1a880a5cdaa]) successfully deleted
W20240506 19:56:43.035689 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8ece0bd3a3b47ef88a0b4910f0efae9 already in progress: creating tablet
W20240506 19:56:43.037196 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet db13b58d17c94e0f82052f823915d9af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db13b58d17c94e0f82052f823915d9af already in progress: creating tablet
W20240506 19:56:43.038060 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ecc88443c04b42d19cc7fba339eeeaf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecc88443c04b42d19cc7fba339eeeaf5 already in progress: creating tablet
I20240506 19:56:43.038656 13246 tablet_replica.cc:330] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:43.039289 13246 raft_consensus.cc:2229] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.039543 13246 raft_consensus.cc:2258] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.040226 13246 ts_tablet_manager.cc:1875] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.040331 13245 tablet_bootstrap.cc:492] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.040480 13245 ts_tablet_manager.cc:1367] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20240506 19:56:43.041368 13245 raft_consensus.cc:348] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.041601 13245 raft_consensus.cc:374] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.041697 13245 raft_consensus.cc:729] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.041955 13245 consensus_queue.cc:260] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.043733 13245 ts_tablet_manager.cc:1398] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:43.044008 13245 tablet_bootstrap.cc:492] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:43.045984 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176c61208b1c4518820a829d9f5c0a6f consensus state has no leader
I20240506 19:56:43.046017 13245 tablet_bootstrap.cc:654] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.046371 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e9155e761b9240aaa949f69e037e6a2b" } new_table_name: "default.Table_a69b09047c6a479789e134aaad2b53ad" modify_external_catalogs: true
I20240506 19:56:43.057533 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:43.058080 12514 catalog_manager.cc:6554] ReplaceTablet: tablet ef03fcaa05d04ebd9f262f43ff7c02e4 deleted and replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e
I20240506 19:56:43.058270 13122 raft_consensus.cc:1261] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:43.058548 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_77892bed220246368648d52f24bdd4df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.059315 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.059535 13353 consensus_queue.cc:1035] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:43.062150 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77892bed220246368648d52f24bdd4df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.064770 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b2a5a5e193f245de96cc82986e292f05
I20240506 19:56:43.065479 13249 tablet_bootstrap.cc:492] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.065616 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a5a5e193f245de96cc82986e292f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:43.065811 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f47449a5a504f939e131743d94433ee consensus state has no leader
I20240506 19:56:43.066957 13249 ts_tablet_manager.cc:1367] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.215s user 0.009s sys 0.000s
I20240506 19:56:43.071681 13249 raft_consensus.cc:348] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.073230 13249 raft_consensus.cc:374] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.074390 13249 raft_consensus.cc:729] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.075584 13245 tablet_bootstrap.cc:492] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.075807 13245 ts_tablet_manager.cc:1367] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20240506 19:56:43.075946 13249 consensus_queue.cc:260] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:43.077594 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e2c9f4baf55412d9bc00818a00c652d consensus state has no leader
I20240506 19:56:43.079381 13246 ts_tablet_manager.cc:1888] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.081059 13249 ts_tablet_manager.cc:1398] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240506 19:56:43.081889 13246 log.cc:1196] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b529bb9362924caca399491e7d9e03cd
I20240506 19:56:43.082370 13246 ts_tablet_manager.cc:1909] T b529bb9362924caca399491e7d9e03cd P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:43.081576 13245 raft_consensus.cc:348] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.083648 13245 raft_consensus.cc:374] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:43.083918 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45da091c44824bd6bb8d390a5fb01ef8 consensus state has no leader
I20240506 19:56:43.083878 13249 tablet_bootstrap.cc:492] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.085121 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 64485c3c13e94055a019f2ef7be1f2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.085412 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.085458 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4ad903e75547fa829b82cc722fef1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.085989 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b529bb9362924caca399491e7d9e03cd (table default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c [id=fd3051399ebf4aa2b31bbd709b120d48]) successfully deleted
I20240506 19:56:43.086755 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a5a5e193f245de96cc82986e292f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.090013 13243 ts_tablet_manager.cc:1888] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:43.091260 13243 log.cc:1196] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/9c18888ae3d6420ca1de3b13fc36cf6c
I20240506 19:56:43.092245 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.095346 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd12a9cc3924f3eb8aec502e9fa2d47 (DEFAULT_TABLE table=default.Table_ca9a4ec248934294830f719bc5d46204 [id=b39d38ddf7f4433fb3fb1ed1c6400a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.095579 13716 raft_consensus.cc:482] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.095754 13716 raft_consensus.cc:504] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.096586 13716 leader_election.cc:288] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:43.097369 12506 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f47449a5a504f939e131743d94433ee consensus state has no leader
I20240506 19:56:43.098052 13243 ts_tablet_manager.cc:1909] T 9c18888ae3d6420ca1de3b13fc36cf6c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:43.099416 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 58c88f80f5b34672b7c3a4b604469bf1 (DEFAULT_TABLE table=default.Table_7714de27e50a4613bb9ad3a6a31b96aa [id=d25fcde9ffaa4b64a0322b921aacc8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.100208 12811 tablet_service.cc:1467] Processing CreateTablet for tablet c5f3def738fb4cb195d08e48d7d2b293 (DEFAULT_TABLE table=default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.101078 12828 tablet_service.cc:1467] Processing CreateTablet for tablet cc22261ab2034e818834ba97c2760dad (DEFAULT_TABLE table=default.Table_319ec87ca4c4468aa4a97b8034323333 [id=5faa78277f6641439177ede005a94e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.101135 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 4216f98cb6ac4e249ca2653518ff9e87 (DEFAULT_TABLE table=default.Table_5553b476197648cd9965dd87bbabb42f [id=00e705e26d69454c980a847fe849a36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.101226 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09cb71b5962f4561bf0fdbe12bf45296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.103017 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd12a9cc3924f3eb8aec502e9fa2d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.103571 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4216f98cb6ac4e249ca2653518ff9e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.103765 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f3def738fb4cb195d08e48d7d2b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.104187 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc22261ab2034e818834ba97c2760dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.104147 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 35453f75a3494f8baf13e0d5cacc9003 (DEFAULT_TABLE table=default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.099359 13244 raft_consensus.cc:348] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.105119 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35453f75a3494f8baf13e0d5cacc9003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.095386 12815 tablet_service.cc:1467] Processing CreateTablet for tablet b6eac98ac8f040c58f61106bb0e989de (DEFAULT_TABLE table=default.Table_4ac15d4e3f6b4719b8b728be5474dd63 [id=15c9ba78bdf345b9b8838a4aeaebb885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.106024 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c88f80f5b34672b7c3a4b604469bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.111280 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca95166c487c4067a47b74eef95deedc" } modify_external_catalogs: true
I20240506 19:56:43.112242 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca95166c487c4067a47b74eef95deedc" } modify_external_catalogs: true
I20240506 19:56:43.117041 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be4e628b9364cccb525b8e1b8070286. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.118582 12781 leader_election.cc:341] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:43.118822 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 378759ca310f453fba45d48879e2dd1a with delete_type TABLET_DATA_DELETED (replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.119068 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 0b899244c35b4bbe9e916c778111f0c0 (DEFAULT_TABLE table=default.Table_dafad97b193f40038dfd516c93b79e96 [id=1a1d40c83b0744689cb8a85164109e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.119650 13244 raft_consensus.cc:374] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.119766 13244 raft_consensus.cc:729] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.120060 13244 consensus_queue.cc:260] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.120632 13244 ts_tablet_manager.cc:1398] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.307s user 0.001s sys 0.001s
I20240506 19:56:43.120977 13244 tablet_bootstrap.cc:492] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:43.125738 13245 raft_consensus.cc:729] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.126178 13245 consensus_queue.cc:260] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.126600 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3291123a1e42af80dbfb578c855618 (DEFAULT_TABLE table=default.Table_012b93e45a4945ee964f19ccd6bbfa0d [id=c30426d9251f45ea93b91e907e358bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.126953 13245 ts_tablet_manager.cc:1398] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.051s user 0.003s sys 0.000s
I20240506 19:56:43.127504 13245 tablet_bootstrap.cc:492] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:43.130564 13245 tablet_bootstrap.cc:654] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.130825 12519 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:43.131987 13246 tablet_replica.cc:330] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:43.132180 13246 raft_consensus.cc:2229] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.132354 13246 raft_consensus.cc:2258] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.132874 13246 ts_tablet_manager.cc:1875] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.132817 12820 tablet_service.cc:1467] Processing CreateTablet for tablet ee69d8e09c2f4c898ce547ec396ca0e6 (DEFAULT_TABLE table=default.Table_942245e6c2a94a96af9206be9765c407 [id=38df61b9f4a641eeb68e44b1eadf7d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.134320 13730 raft_consensus.cc:482] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.134460 13730 raft_consensus.cc:504] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.135200 13730 leader_election.cc:288] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:43.136925 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 8321457717fd4ad28ae1292d9e9a8669 (DEFAULT_TABLE table=default.Table_109dfbf34cd845b3b626bc0a00a23e52 [id=45a5514e39a44c6a993cf99f4a566fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.137094 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6eac98ac8f040c58f61106bb0e989de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.137480 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8321457717fd4ad28ae1292d9e9a8669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.139417 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4ad903e75547fa829b82cc722fef1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.139698 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa (DEFAULT_TABLE table=default.Table_4674aa2f59294605bc41516bbae12158 [id=ea63555353cd45d485b18c833792fda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.141032 13095 tablet_service.cc:1467] Processing CreateTablet for tablet d19db9cff74843e087fd82db2475c29c (DEFAULT_TABLE table=default.Table_138cdbf4ae874f409ca3414d505a1217 [id=6d36a94991d648548bbc2d86b227b1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.141487 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19db9cff74843e087fd82db2475c29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.141532 12819 tablet_service.cc:1467] Processing CreateTablet for tablet e6d37bb1c2144c4fa71aab6113b9f06e (DEFAULT_TABLE table=default.Table_195d90ff34ff4ca6b6282483c9691f47 [id=99717b6265bc402089da1d146125f883]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.142833 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb8bb09098d401da62ad737ffb697ad (DEFAULT_TABLE table=default.Table_373183be2430417d8304f5e10cc72a57 [id=65fbfa6971874ba9a9f388961010594e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.143046 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b899244c35b4bbe9e916c778111f0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.141860 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 64485c3c13e94055a019f2ef7be1f2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.143646 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a5a5e193f245de96cc82986e292f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.143846 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 116df9bd6ab04a30ba09861e765d7267 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.143952 13716 raft_consensus.cc:482] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.144785 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.145284 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea25fb8e42234b82885b8fb2275d4925" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.145900 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 8321457717fd4ad28ae1292d9e9a8669 (DEFAULT_TABLE table=default.Table_109dfbf34cd845b3b626bc0a00a23e52 [id=45a5514e39a44c6a993cf99f4a566fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.145917 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb1559fe135498f9fefbbf04daa6909 (DEFAULT_TABLE table=default.Table_7604b11105d14a889c9b95c21133e402 [id=617fd0a3b26f40abaee4b419ead18cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.146320 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb1559fe135498f9fefbbf04daa6909. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.146502 13058 leader_election.cc:341] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.148450 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a080ccac6394210b0b24b65843f9d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.148821 12816 tablet_service.cc:1467] Processing CreateTablet for tablet d19db9cff74843e087fd82db2475c29c (DEFAULT_TABLE table=default.Table_138cdbf4ae874f409ca3414d505a1217 [id=6d36a94991d648548bbc2d86b227b1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.149320 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb1559fe135498f9fefbbf04daa6909 (DEFAULT_TABLE table=default.Table_7604b11105d14a889c9b95c21133e402 [id=617fd0a3b26f40abaee4b419ead18cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.149812 13104 tablet_service.cc:1467] Processing CreateTablet for tablet e02577dc6c2049a1abb47ed0fe729e18 (DEFAULT_TABLE table=default.Table_001c9250e65b432f8e9d4c1a38bda161 [id=5c1e34eaaf154ca286d558bc58aef267]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.150241 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02577dc6c2049a1abb47ed0fe729e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.153445 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea8f5301f524e4e8910ce61837a1dd4 (DEFAULT_TABLE table=default.Table_300911c21c1140aca1cd2170583017fb [id=6aecb90966cd41f9aa1066cd32246615]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.153903 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea8f5301f524e4e8910ce61837a1dd4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.157964 12483 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu pid 12630: Timed out: CheckLeaks RPC to 127.12.48.252:36867 timed out after 1.000s (SENT)
I20240506 19:56:43.158161 12483 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu with pid 12630
I20240506 19:56:43.158332 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 31c440555cfe4195b3ae86e1f1eb6a34
I20240506 19:56:43.160072 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.162920 13089 tablet_service.cc:1467] Processing CreateTablet for tablet c652feb405ef4f1cab8ee602e9d3eb18 (DEFAULT_TABLE table=default.Table_cddf9ad090cc4ac690a9a68b8600791e [id=9df40a8cbb4548d6a4edb96b3d99bfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.163376 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c652feb405ef4f1cab8ee602e9d3eb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.168586 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 65da8b15e0c6496bb86b47c49e348af0 (DEFAULT_TABLE table=default.Table_67154b63102e41e0b35410affe262e62 [id=e1d60ac8788c479881332deabab8cd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.169106 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da8b15e0c6496bb86b47c49e348af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.176275 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 9c19567619b64ee0a8dd637d7f69ce3e (DEFAULT_TABLE table=default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 [id=1880f76f4434494090340781bfef83ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.183032 13095 tablet_service.cc:1467] Processing CreateTablet for tablet f63e8e12fd3b453a96413f1fc769fb9f (DEFAULT_TABLE table=default.Table_bca67f775cc9457cb0bbcd9b659322e8 [id=95ada67289714d7d8529b5338c361fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.187269 13089 tablet_service.cc:1467] Processing CreateTablet for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 (DEFAULT_TABLE table=default.Table_1e8cbfa77d85462baa8b75f47a5c76bf [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.191344 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c19567619b64ee0a8dd637d7f69ce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.191776 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63e8e12fd3b453a96413f1fc769fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.192530 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.199729 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:43.199955 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d37bb1c2144c4fa71aab6113b9f06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.200052 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:43.200322 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.200608 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8321457717fd4ad28ae1292d9e9a8669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.200881 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb8bb09098d401da62ad737ffb697ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.203218 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet a36b4d51b100486c831a41a52d81fd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:43.204241 12783 leader_election.cc:341] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:43.204828 13835 tablet.cc:1722] T e1f1e4efbe8b40b199fcb3020569e1a1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:43.205965 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d95065af2929456c8ae9740f3fcffcdc" } modify_external_catalogs: true
I20240506 19:56:43.206226 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d95065af2929456c8ae9740f3fcffcdc" } modify_external_catalogs: true
I20240506 19:56:43.206858 13716 raft_consensus.cc:504] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.208774 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a9407b54478a4b51b62ed049b31e30bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.209386 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9407b54478a4b51b62ed049b31e30bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.210870 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_de5cf031a4074ce78c0207432422ae66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.211416 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de5cf031a4074ce78c0207432422ae66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.212675 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9e3f4bda34084998998a5a34a7717ffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.213148 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e3f4bda34084998998a5a34a7717ffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.213521 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet ea25fb8e42234b82885b8fb2275d4925 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.214635 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee1a4a3f060407ca9c33467474e4489 (DEFAULT_TABLE table=default.Table_85795c658c664939838d720f20e0477a [id=28724ef5aa3b4dc2b399336e8959cf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.216734 13716 leader_election.cc:288] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:43.218014 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee1a4a3f060407ca9c33467474e4489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.218166 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]) successfully deleted
W20240506 19:56:43.219084 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 378759ca310f453fba45d48879e2dd1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 378759ca310f453fba45d48879e2dd1a already in progress: creating tablet
W20240506 19:56:43.106022 12921 consensus_peers.cc:485] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:43.219560 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 113651us
I20240506 19:56:43.107113 13629 raft_consensus.cc:482] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.219842 13629 raft_consensus.cc:504] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.135875 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 8321457717fd4ad28ae1292d9e9a8669 (DEFAULT_TABLE table=default.Table_109dfbf34cd845b3b626bc0a00a23e52 [id=45a5514e39a44c6a993cf99f4a566fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.220352 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8321457717fd4ad28ae1292d9e9a8669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.220515 13629 leader_election.cc:288] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:43.136083 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea25fb8e42234b82885b8fb2275d4925" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:43.221588 12986 raft_consensus.cc:2454] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:43.221956 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0114edac976844619e438c17bb959010" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.160401 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.148746 12963 tablet_service.cc:1467] Processing CreateTablet for tablet e02577dc6c2049a1abb47ed0fe729e18 (DEFAULT_TABLE table=default.Table_001c9250e65b432f8e9d4c1a38bda161 [id=5c1e34eaaf154ca286d558bc58aef267]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.161969 12954 tablet_service.cc:1467] Processing CreateTablet for tablet c652feb405ef4f1cab8ee602e9d3eb18 (DEFAULT_TABLE table=default.Table_cddf9ad090cc4ac690a9a68b8600791e [id=9df40a8cbb4548d6a4edb96b3d99bfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.164956 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 65da8b15e0c6496bb86b47c49e348af0 (DEFAULT_TABLE table=default.Table_67154b63102e41e0b35410affe262e62 [id=e1d60ac8788c479881332deabab8cd25]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:43.222572 12921 leader_election.cc:341] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.102463 13249 tablet_bootstrap.cc:654] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.172508 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee1a4a3f060407ca9c33467474e4489 (DEFAULT_TABLE table=default.Table_85795c658c664939838d720f20e0477a [id=28724ef5aa3b4dc2b399336e8959cf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.175022 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 9c19567619b64ee0a8dd637d7f69ce3e (DEFAULT_TABLE table=default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 [id=1880f76f4434494090340781bfef83ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.180991 12952 tablet_service.cc:1467] Processing CreateTablet for tablet f63e8e12fd3b453a96413f1fc769fb9f (DEFAULT_TABLE table=default.Table_bca67f775cc9457cb0bbcd9b659322e8 [id=95ada67289714d7d8529b5338c361fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.223922 12783 leader_election.cc:302] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:43.186332 12964 tablet_service.cc:1467] Processing CreateTablet for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 (DEFAULT_TABLE table=default.Table_1e8cbfa77d85462baa8b75f47a5c76bf [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.228222 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb1559fe135498f9fefbbf04daa6909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.229383 13716 raft_consensus.cc:2726] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.138847 12962 tablet_service.cc:1467] Processing CreateTablet for tablet d19db9cff74843e087fd82db2475c29c (DEFAULT_TABLE table=default.Table_138cdbf4ae874f409ca3414d505a1217 [id=6d36a94991d648548bbc2d86b227b1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.229758 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee69d8e09c2f4c898ce547ec396ca0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.144183 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb1559fe135498f9fefbbf04daa6909 (DEFAULT_TABLE table=default.Table_7604b11105d14a889c9b95c21133e402 [id=617fd0a3b26f40abaee4b419ead18cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.151988 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea8f5301f524e4e8910ce61837a1dd4 (DEFAULT_TABLE table=default.Table_300911c21c1140aca1cd2170583017fb [id=6aecb90966cd41f9aa1066cd32246615]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.107378 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.184296 13834 tablet.cc:1722] T e1f1e4efbe8b40b199fcb3020569e1a1 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:43.235242 13730 raft_consensus.cc:482] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.219400 13243 tablet_replica.cc:330] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:43.235440 13730 raft_consensus.cc:504] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.235589 13243 raft_consensus.cc:2229] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.235781 13243 raft_consensus.cc:2258] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.235834 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19db9cff74843e087fd82db2475c29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.236155 13730 leader_election.cc:288] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:43.237619 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:43.238371 13058 leader_election.cc:341] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.241043 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_768e769dac9046d582bf4c77e1221d9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.241333 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3291123a1e42af80dbfb578c855618. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.241704 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_768e769dac9046d582bf4c77e1221d9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.242403 13245 tablet_bootstrap.cc:492] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.243086 13245 ts_tablet_manager.cc:1367] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.001s
I20240506 19:56:43.243059 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_77892bed220246368648d52f24bdd4df" } modify_external_catalogs: true
I20240506 19:56:43.243269 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_77892bed220246368648d52f24bdd4df" } modify_external_catalogs: true
I20240506 19:56:43.244616 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aeb68c202cc642e6908287b378e38344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.245182 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeb68c202cc642e6908287b378e38344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.246558 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_72cbb579f8c5457ab8ef64876959a3a5" } modify_external_catalogs: true
I20240506 19:56:43.246752 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_72cbb579f8c5457ab8ef64876959a3a5" } modify_external_catalogs: true
I20240506 19:56:43.248776 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 59b022c086d141a19a9b35c4a8c8bef9
I20240506 19:56:43.250283 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_86f15af99695455a9f33070a0bc8563f" } new_table_name: "default.Table_8179008915b24338af37dcab927fc4e8" modify_external_catalogs: true
W20240506 19:56:43.252669 12783 leader_election.cc:341] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 0ba2365d00044c32a06f9017a207adf7
I20240506 19:56:43.253896 13245 raft_consensus.cc:348] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.254169 13245 raft_consensus.cc:374] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.254112 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7b7fe87d3acd4184b95b2dd1026980d5" } new_table_name: "default.Table_fd8d81f3e15442a4926979026de42b20" modify_external_catalogs: true
I20240506 19:56:43.254264 13245 raft_consensus.cc:729] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.258116 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b022c086d141a19a9b35c4a8c8bef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.262037 13245 consensus_queue.cc:260] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.263083 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b022c086d141a19a9b35c4a8c8bef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:43.265843 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85365758f6a64536ab61e8349de63dea consensus state has no leader
I20240506 19:56:43.266381 12517 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 5a3ee988f10b4842947786f45163ec8e
I20240506 19:56:43.270494 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.274647 12822 tablet_service.cc:1467] Processing CreateTablet for tablet e02577dc6c2049a1abb47ed0fe729e18 (DEFAULT_TABLE table=default.Table_001c9250e65b432f8e9d4c1a38bda161 [id=5c1e34eaaf154ca286d558bc58aef267]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.276087 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9f13287a26d24d28813ef3727d8e6844"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.276736 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f13287a26d24d28813ef3727d8e6844 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.277500 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ee7192b3b36648858181f492bf1a72b9" } modify_external_catalogs: true
I20240506 19:56:43.277660 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ee7192b3b36648858181f492bf1a72b9" } modify_external_catalogs: true
I20240506 19:56:43.280923 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8507072bb8ea467982eab99c978a7ac8" } new_table_name: "default.Table_3e8858530f1441ffa975a8576f0ffd5b" modify_external_catalogs: true
I20240506 19:56:43.281530 13243 ts_tablet_manager.cc:1875] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.282431 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02577dc6c2049a1abb47ed0fe729e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.282694 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.283133 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a24848c14dfc4d0ca604aabf57ec28d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.283717 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a24848c14dfc4d0ca604aabf57ec28d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.285346 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_73585d09b2764941be3d724930ffd097"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.285810 13245 ts_tablet_manager.cc:1398] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
W20240506 19:56:43.285918 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73585d09b2764941be3d724930ffd097 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.287668 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3612dd9500444861ac651d75a8b106a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.288249 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3612dd9500444861ac651d75a8b106a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.289407 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea8f5301f524e4e8910ce61837a1dd4 (DEFAULT_TABLE table=default.Table_300911c21c1140aca1cd2170583017fb [id=6aecb90966cd41f9aa1066cd32246615]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.293184 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4af9a240535b494fbe65842e4ddc343b" } new_table_name: "default.Table_fc5383e806314718975782b69f0243ed" modify_external_catalogs: true
I20240506 19:56:43.293416 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19db9cff74843e087fd82db2475c29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.294405 13245 tablet_bootstrap.cc:492] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:43.296674 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fed2f12eda54be0879b1e9c33ca3e31 consensus state has no leader
I20240506 19:56:43.297490 13245 tablet_bootstrap.cc:654] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.298041 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:43.301093 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c19567619b64ee0a8dd637d7f69ce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.298962 13246 ts_tablet_manager.cc:1888] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.301967 13246 log.cc:1196] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/5988b89cb52e4f858b52862138bed025
I20240506 19:56:43.302738 13246 ts_tablet_manager.cc:1909] T 5988b89cb52e4f858b52862138bed025 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:43.311717 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 5988b89cb52e4f858b52862138bed025 (table default.Table_7ec0af39e86c4ffeb0622785e6a817be [id=cd2382012b444dd09cd9ad1a50a20b4c]) successfully deleted
I20240506 19:56:43.312420 13716 raft_consensus.cc:482] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.314571 13730 raft_consensus.cc:482] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.314764 13730 raft_consensus.cc:504] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.315501 13730 leader_election.cc:288] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:43.317914 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63e8e12fd3b453a96413f1fc769fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.318924 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea8f5301f524e4e8910ce61837a1dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.321420 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea8f5301f524e4e8910ce61837a1dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.324864 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da8b15e0c6496bb86b47c49e348af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.325613 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee1a4a3f060407ca9c33467474e4489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.328516 13716 raft_consensus.cc:504] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.331678 13242 ts_tablet_manager.cc:1888] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240506 19:56:43.334213 13246 tablet_replica.cc:330] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:43.335467 13249 tablet_bootstrap.cc:492] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
W20240506 19:56:43.335922 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d9443cdbaa3c40299b66d216673c4b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9443cdbaa3c40299b66d216673c4b1a already in progress: creating tablet
I20240506 19:56:43.336658 13249 ts_tablet_manager.cc:1367] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.254s user 0.004s sys 0.008s
I20240506 19:56:43.337606 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.338492 13246 raft_consensus.cc:2229] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.338757 13246 raft_consensus.cc:2258] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.339461 13246 ts_tablet_manager.cc:1875] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.340924 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb1559fe135498f9fefbbf04daa6909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.341238 13837 raft_consensus.cc:482] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.341455 13837 raft_consensus.cc:504] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.342903 13837 leader_election.cc:288] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:43.343274 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c652feb405ef4f1cab8ee602e9d3eb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.345654 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02577dc6c2049a1abb47ed0fe729e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.351977 12825 tablet_service.cc:1467] Processing CreateTablet for tablet c652feb405ef4f1cab8ee602e9d3eb18 (DEFAULT_TABLE table=default.Table_cddf9ad090cc4ac690a9a68b8600791e [id=9df40a8cbb4548d6a4edb96b3d99bfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.352507 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c652feb405ef4f1cab8ee602e9d3eb18. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.357637 13058 connection.cc:535] client connection to 127.12.48.252:36867 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:56:43.358161 12507 connection.cc:535] client connection to 127.12.48.252:36867 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:56:43.358361 12507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 4 similar messages]
I20240506 19:56:43.360627 13245 tablet_bootstrap.cc:492] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.360921 13245 ts_tablet_manager.cc:1367] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.005s
I20240506 19:56:43.366562 13249 raft_consensus.cc:348] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.368395 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 65da8b15e0c6496bb86b47c49e348af0 (DEFAULT_TABLE table=default.Table_67154b63102e41e0b35410affe262e62 [id=e1d60ac8788c479881332deabab8cd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.361860 13245 raft_consensus.cc:348] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.369498 13242 log.cc:1196] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/6156c6f582b747de973b357b6d071a46
W20240506 19:56:43.369705 12782 connection.cc:535] client connection to 127.12.48.252:36867 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 19:56:43.370889 13245 raft_consensus.cc:374] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.371497 13245 raft_consensus.cc:729] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.372495 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee1a4a3f060407ca9c33467474e4489 (DEFAULT_TABLE table=default.Table_85795c658c664939838d720f20e0477a [id=28724ef5aa3b4dc2b399336e8959cf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.373749 13245 consensus_queue.cc:260] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.375387 13716 leader_election.cc:288] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:43.377189 12783 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 30d819d1237f4798b4847fdf378cbc62
W20240506 19:56:43.366256 12921 connection.cc:535] client connection to 127.12.48.252:36867 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 19:56:43.383735 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8179008915b24338af37dcab927fc4e8" } new_table_name: "default.Table_58aa959f517f46b6a0c3a485458211bd" modify_external_catalogs: true
I20240506 19:56:43.385545 12520 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3] alter complete (version 3)
I20240506 19:56:43.386263 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da8b15e0c6496bb86b47c49e348af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.386348 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fcad4ec094e649709cb1cf84577e7483"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.387169 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcad4ec094e649709cb1cf84577e7483 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.389222 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2e36b2f9be7f4af9a58215847e671f2f" } modify_external_catalogs: true
I20240506 19:56:43.389484 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2e36b2f9be7f4af9a58215847e671f2f" } modify_external_catalogs: true
I20240506 19:56:43.390007 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 9c19567619b64ee0a8dd637d7f69ce3e (DEFAULT_TABLE table=default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 [id=1880f76f4434494090340781bfef83ba]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:43.396209 12783 consensus_peers.cc:485] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2502b423bd1040539fb1a3ebbece3c1c. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:43.398046 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c19567619b64ee0a8dd637d7f69ce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.400017 13242 ts_tablet_manager.cc:1909] T 6156c6f582b747de973b357b6d071a46 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:43.402182 13249 raft_consensus.cc:374] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.405427 13244 tablet_bootstrap.cc:654] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.407904 13245 ts_tablet_manager.cc:1398] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20240506 19:56:43.408739 13249 raft_consensus.cc:729] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.413708 13249 consensus_queue.cc:260] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:43.414521 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 225580us
I20240506 19:56:43.416801 13057 leader_election.cc:302] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:43.417845 13730 raft_consensus.cc:2781] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:43.419834 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:43.423804 13730 raft_consensus.cc:482] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:43.423966 13730 raft_consensus.cc:3037] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:43.424465 12814 tablet_service.cc:1467] Processing CreateTablet for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 (DEFAULT_TABLE table=default.Table_1e8cbfa77d85462baa8b75f47a5c76bf [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.427088 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0114edac976844619e438c17bb959010" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:43.429123 13730 raft_consensus.cc:504] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.430349 13730 leader_election.cc:288] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:43.431456 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c56ef126a4ca48b6a379a6771bc1c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:43.432651 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea25fb8e42234b82885b8fb2275d4925" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:43.433369 13249 ts_tablet_manager.cc:1398] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.079s user 0.000s sys 0.002s
I20240506 19:56:43.434401 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0114edac976844619e438c17bb959010" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:43.434214 13058 leader_election.cc:341] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.434731 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0114edac976844619e438c17bb959010" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.435878 13243 ts_tablet_manager.cc:1888] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:43.435994 13058 leader_election.cc:341] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.436058 13243 log.cc:1196] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/0114edac976844619e438c17bb959010
I20240506 19:56:43.436317 13249 tablet_bootstrap.cc:492] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.437855 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 386) took 2650 ms. Trace:
I20240506 19:56:43.438163 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee1a4a3f060407ca9c33467474e4489. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.438606 12922 leader_election.cc:341] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:43.438825 12922 leader_election.cc:302] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:43.439354 12922 raft_consensus.cc:2670] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240506 19:56:43.441422 12922 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:43.442143 13057 leader_election.cc:341] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.442266 13057 leader_election.cc:302] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:43.442561 13730 raft_consensus.cc:2726] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.441679 13838 consensus_queue.cc:1035] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:43.443302 13243 ts_tablet_manager.cc:1909] T 0114edac976844619e438c17bb959010 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:43.444345 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 0114edac976844619e438c17bb959010 (table default.Table_ac829ce2695e44849b1d44918904dc8a [id=c2803598df8241beb0b6ce684515661c]) successfully deleted
W20240506 19:56:43.444909 13057 leader_election.cc:341] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:43.445027 13057 leader_election.cc:302] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:43.445204 13057 raft_consensus.cc:2670] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:43.445921 13249 tablet_bootstrap.cc:654] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.449586 12828 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b022c086d141a19a9b35c4a8c8bef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.449965 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea25fb8e42234b82885b8fb2275d4925" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:43.450196 12986 raft_consensus.cc:3037] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:43.452380 12811 tablet_service.cc:1467] Processing CreateTablet for tablet f63e8e12fd3b453a96413f1fc769fb9f (DEFAULT_TABLE table=default.Table_bca67f775cc9457cb0bbcd9b659322e8 [id=95ada67289714d7d8529b5338c361fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.453753 12986 raft_consensus.cc:2454] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:43.454532 13057 leader_election.cc:302] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:43.455214 13730 raft_consensus.cc:2781] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:43.461004 13243 tablet_replica.cc:330] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:43.461911 13243 raft_consensus.cc:2229] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.462178 13243 raft_consensus.cc:2258] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.462957 13243 ts_tablet_manager.cc:1875] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.464210 13242 rpcz_store.cc:276] 0506 19:56:40.755100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.755167 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:42.775680 (+2020513us) ts_tablet_manager.cc:1120] Deleting tablet 6156c6f582b747de973b357b6d071a46
0506 19:56:42.775687 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:42.812964 (+ 37277us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:43.331530 (+518566us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:43.405913 (+ 74383us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4678,"tablet-delete.queue_time_us":2020147}
I20240506 19:56:43.473484 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.476213 13245 tablet_bootstrap.cc:492] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:43.477213 13242 tablet_replica.cc:330] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:43.478135 13245 tablet_bootstrap.cc:654] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.478403 13249 tablet_bootstrap.cc:492] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.478739 13249 ts_tablet_manager.cc:1367] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.003s
I20240506 19:56:43.479862 13249 raft_consensus.cc:348] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.480278 13249 raft_consensus.cc:374] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.480419 13249 raft_consensus.cc:729] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.480896 13249 consensus_queue.cc:260] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.483295 13249 ts_tablet_manager.cc:1398] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:43.485822 13249 tablet_bootstrap.cc:492] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.488165 13246 ts_tablet_manager.cc:1888] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.488405 13246 log.cc:1196] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/0114edac976844619e438c17bb959010
I20240506 19:56:43.488629 13249 tablet_bootstrap.cc:654] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.488847 13246 ts_tablet_manager.cc:1909] T 0114edac976844619e438c17bb959010 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:43.495182 13242 raft_consensus.cc:2229] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.495496 13242 raft_consensus.cc:2258] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.492935 13243 ts_tablet_manager.cc:1888] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.499372 13243 log.cc:1196] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:43.499917 13243 ts_tablet_manager.cc:1909] T ca394b7f53ce445e8778dd25d0cf25c7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:43.501680 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ca394b7f53ce445e8778dd25d0cf25c7 (table default.Table_ca61f3a9aefa4fba8fea32d717de3fb1 [id=bf75c039ad324e50a2ccb889439e2b4c]) successfully deleted
W20240506 19:56:43.502765 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 116df9bd6ab04a30ba09861e765d7267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 116df9bd6ab04a30ba09861e765d7267 already in progress: creating tablet
I20240506 19:56:43.503444 13243 tablet_replica.cc:330] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:43.503083 13242 ts_tablet_manager.cc:1875] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.504071 13243 raft_consensus.cc:2229] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.504299 13243 raft_consensus.cc:2258] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.506672 13730 raft_consensus.cc:686] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:43.507179 13730 consensus_queue.cc:237] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.507702 13243 ts_tablet_manager.cc:1875] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.511308 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63e8e12fd3b453a96413f1fc769fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.514968 13249 tablet_bootstrap.cc:492] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.515266 13249 ts_tablet_manager.cc:1367] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20240506 19:56:43.516434 13249 raft_consensus.cc:348] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.518510 13249 raft_consensus.cc:374] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.518723 13249 raft_consensus.cc:729] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.519142 13249 consensus_queue.cc:260] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.522481 13246 tablet_replica.cc:330] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:43.522773 13246 raft_consensus.cc:2229] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.523052 13246 raft_consensus.cc:2258] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.523821 13246 ts_tablet_manager.cc:1875] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.524652 13249 ts_tablet_manager.cc:1398] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:43.524778 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 0114edac976844619e438c17bb959010 (table default.Table_ac829ce2695e44849b1d44918904dc8a [id=c2803598df8241beb0b6ce684515661c]) successfully deleted
I20240506 19:56:43.526317 13249 tablet_bootstrap.cc:492] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.529903 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 6156c6f582b747de973b357b6d071a46 (table default.Table_44e570af6caa4a749207a8c828639acf [id=ee08d3829f304e16b5b8994259bb11b0]) successfully deleted
I20240506 19:56:43.530069 13249 tablet_bootstrap.cc:654] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:43.532497 12507 consensus_peers.cc:485] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d -> Peer 97f390a786034cddbff757779037f861 (127.12.48.252:36867): Couldn't send request to peer 97f390a786034cddbff757779037f861. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:43.535933 13848 raft_consensus.cc:482] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.536149 13848 raft_consensus.cc:504] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.537683 13848 leader_election.cc:288] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:43.538221 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cade6b1c5a034b23aa33b2349e1dab95 consensus state has no leader
I20240506 19:56:43.538667 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6d22301588d54dcd8649bc08a1ade8f6" } modify_external_catalogs: true
I20240506 19:56:43.538885 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6d22301588d54dcd8649bc08a1ade8f6" } modify_external_catalogs: true
I20240506 19:56:43.541098 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:43.541715 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45a853e310473d844a423fbefc553a (DEFAULT_TABLE table=default.Table_ee7192b3b36648858181f492bf1a72b9 [id=f1e89032cff9464789c466bef5d11c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.542166 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45a853e310473d844a423fbefc553a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.542938 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad7076ed00f4aec9cb92ba7860932b6 (DEFAULT_TABLE table=default.Table_50e680b80e9e48faa1d0e0e07bf41693 [id=37f5cb0fe4b348e4a1e0a9d4cec538be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.543327 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad7076ed00f4aec9cb92ba7860932b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.544392 13057 leader_election.cc:341] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.544823 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45a853e310473d844a423fbefc553a (DEFAULT_TABLE table=default.Table_ee7192b3b36648858181f492bf1a72b9 [id=f1e89032cff9464789c466bef5d11c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.546227 13097 tablet_service.cc:1467] Processing CreateTablet for tablet dc2c509a06554b9db6d5e0be97e159db (DEFAULT_TABLE table=default.Table_2b9776133219459d8dfd0ea02428d80f [id=17e3ad30ddd346be912bdb5ebfecdb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.546531 13089 tablet_service.cc:1467] Processing CreateTablet for tablet a18ad478630e491cab700ac0c42a5ede (DEFAULT_TABLE table=default.Table_1aacc41134ef4e40ac2c05d45c1faf3a [id=1a82494c8acb4e948297879f4b780060]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.546725 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc2c509a06554b9db6d5e0be97e159db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.547000 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a18ad478630e491cab700ac0c42a5ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.547389 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 902d43d15d334939bddb254ae04ca174 (DEFAULT_TABLE table=default.Table_f249dc790e774c46a2cf5e2af4f2a7a5 [id=f48f19c2b56e498dbdd65515f355c419]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.547878 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902d43d15d334939bddb254ae04ca174. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.545276 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45a853e310473d844a423fbefc553a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.554179 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 902d43d15d334939bddb254ae04ca174 (DEFAULT_TABLE table=default.Table_f249dc790e774c46a2cf5e2af4f2a7a5 [id=f48f19c2b56e498dbdd65515f355c419]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.554667 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902d43d15d334939bddb254ae04ca174. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.545584 12962 tablet_service.cc:1467] Processing CreateTablet for tablet dc2c509a06554b9db6d5e0be97e159db (DEFAULT_TABLE table=default.Table_2b9776133219459d8dfd0ea02428d80f [id=17e3ad30ddd346be912bdb5ebfecdb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.556732 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.557075 13245 tablet_bootstrap.cc:492] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.557242 13245 ts_tablet_manager.cc:1367] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.001s
W20240506 19:56:43.558250 13058 leader_election.cc:341] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.558410 13058 leader_election.cc:302] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:43.558480 13245 raft_consensus.cc:348] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.558686 13245 raft_consensus.cc:374] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.558796 13245 raft_consensus.cc:729] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.559028 13245 consensus_queue.cc:260] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:43.561554 12781 leader_election.cc:341] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 0ba2365d00044c32a06f9017a207adf7
I20240506 19:56:43.562636 13849 raft_consensus.cc:2726] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.563099 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 902d43d15d334939bddb254ae04ca174 (DEFAULT_TABLE table=default.Table_f249dc790e774c46a2cf5e2af4f2a7a5 [id=f48f19c2b56e498dbdd65515f355c419]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.563226 12816 tablet_service.cc:1467] Processing CreateTablet for tablet dc2c509a06554b9db6d5e0be97e159db (DEFAULT_TABLE table=default.Table_2b9776133219459d8dfd0ea02428d80f [id=17e3ad30ddd346be912bdb5ebfecdb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.564069 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc2c509a06554b9db6d5e0be97e159db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.564689 12949 tablet_service.cc:1467] Processing CreateTablet for tablet a18ad478630e491cab700ac0c42a5ede (DEFAULT_TABLE table=default.Table_1aacc41134ef4e40ac2c05d45c1faf3a [id=1a82494c8acb4e948297879f4b780060]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.564924 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7881939105214872a57c956985a14360"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.565573 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7881939105214872a57c956985a14360 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:43.566831 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1b011e94e2b47d7a51d6816576ea124 consensus state has no leader
I20240506 19:56:43.568013 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 6097fe368a894f648b097912e16e361f (DEFAULT_TABLE table=default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8 [id=f5816273b8914052a9b58d5495bf45a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.568132 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a18ad478630e491cab700ac0c42a5ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.568485 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6097fe368a894f648b097912e16e361f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.568464 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_538bc8e0e1604e09bed86b488cf7ada0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.569023 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_538bc8e0e1604e09bed86b488cf7ada0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.570816 12948 tablet_service.cc:1467] Processing CreateTablet for tablet dc206b31d3da458cb944d5f00e23d702 (DEFAULT_TABLE table=default.Table_88a94bbd43f4495aac452adfdfa73c5d [id=cf25820d73f248818c5db30762266132]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.571419 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad7076ed00f4aec9cb92ba7860932b6 (DEFAULT_TABLE table=default.Table_50e680b80e9e48faa1d0e0e07bf41693 [id=37f5cb0fe4b348e4a1e0a9d4cec538be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.571689 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc56299ed854d6c8ef201127692fd60 (DEFAULT_TABLE table=default.Table_ea8710b001ee40d9aec404921fb4dcd3 [id=5fd35e6c8f3d43b0a86ac995dc55762f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.571871 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet a36b4d51b100486c831a41a52d81fd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.572342 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc206b31d3da458cb944d5f00e23d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.573105 13085 tablet_service.cc:1467] Processing CreateTablet for tablet dc206b31d3da458cb944d5f00e23d702 (DEFAULT_TABLE table=default.Table_88a94bbd43f4495aac452adfdfa73c5d [id=cf25820d73f248818c5db30762266132]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.573596 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc206b31d3da458cb944d5f00e23d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.574379 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc56299ed854d6c8ef201127692fd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.576267 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad7076ed00f4aec9cb92ba7860932b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.582373 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 6097fe368a894f648b097912e16e361f (DEFAULT_TABLE table=default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8 [id=f5816273b8914052a9b58d5495bf45a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.577986 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad7076ed00f4aec9cb92ba7860932b6 (DEFAULT_TABLE table=default.Table_50e680b80e9e48faa1d0e0e07bf41693 [id=37f5cb0fe4b348e4a1e0a9d4cec538be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.582995 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6097fe368a894f648b097912e16e361f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.583091 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 6097fe368a894f648b097912e16e361f (DEFAULT_TABLE table=default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8 [id=f5816273b8914052a9b58d5495bf45a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.584069 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc56299ed854d6c8ef201127692fd60 (DEFAULT_TABLE table=default.Table_ea8710b001ee40d9aec404921fb4dcd3 [id=5fd35e6c8f3d43b0a86ac995dc55762f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.581427 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0f727575df874202b2cdd6ca62d5727f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.591320 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f727575df874202b2cdd6ca62d5727f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.594018 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc30f4690b24c5ea8ab2f7a26122992 (DEFAULT_TABLE table=default.Table_6f7629baef0946e6aa92b1cbdef4d362 [id=e57f28ac2f0e4ad49895a608f139c0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.594332 13245 ts_tablet_manager.cc:1398] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20240506 19:56:43.602016 13245 tablet_bootstrap.cc:492] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:43.598049 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc56299ed854d6c8ef201127692fd60 (DEFAULT_TABLE table=default.Table_ea8710b001ee40d9aec404921fb4dcd3 [id=5fd35e6c8f3d43b0a86ac995dc55762f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.620043 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45a853e310473d844a423fbefc553a (DEFAULT_TABLE table=default.Table_ee7192b3b36648858181f492bf1a72b9 [id=f1e89032cff9464789c466bef5d11c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.620501 12822 tablet_service.cc:1467] Processing CreateTablet for tablet a18ad478630e491cab700ac0c42a5ede (DEFAULT_TABLE table=default.Table_1aacc41134ef4e40ac2c05d45c1faf3a [id=1a82494c8acb4e948297879f4b780060]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.621505 12519 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet aae313961a914ed485f866d85785a788 (table default.Table_cc5c3f3202074991b8db4134d5824ad4 [id=a63e1c9e058a4501831a67516affe578]). Expected version 1 got 0
I20240506 19:56:43.623517 13244 tablet_bootstrap.cc:492] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:43.623680 13244 ts_tablet_manager.cc:1367] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.503s user 0.012s sys 0.005s
I20240506 19:56:43.624989 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7984f6e0fe485fa0c15fd13c3f428e (DEFAULT_TABLE table=default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.625082 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7984f6e0fe485fa0c15fd13c3f428e (DEFAULT_TABLE table=default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.625643 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc30f4690b24c5ea8ab2f7a26122992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.626046 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc30f4690b24c5ea8ab2f7a26122992 (DEFAULT_TABLE table=default.Table_6f7629baef0946e6aa92b1cbdef4d362 [id=e57f28ac2f0e4ad49895a608f139c0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.626708 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_de5cf031a4074ce78c0207432422ae66" } new_table_name: "default.Table_f8a1ac708a7d4f20bb111b87031710c9" modify_external_catalogs: true
I20240506 19:56:43.627643 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902d43d15d334939bddb254ae04ca174. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.617759 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc56299ed854d6c8ef201127692fd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.633461 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7984f6e0fe485fa0c15fd13c3f428e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.641965 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a18ad478630e491cab700ac0c42a5ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.644035 12809 tablet_service.cc:1467] Processing CreateTablet for tablet dc206b31d3da458cb944d5f00e23d702 (DEFAULT_TABLE table=default.Table_88a94bbd43f4495aac452adfdfa73c5d [id=cf25820d73f248818c5db30762266132]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.644667 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc30f4690b24c5ea8ab2f7a26122992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.644865 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc206b31d3da458cb944d5f00e23d702. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:43.648226 12783 consensus_peers.cc:485] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5725b47e43244f0af881b07e145b3aa. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:43.642560 13244 raft_consensus.cc:348] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.648468 13244 raft_consensus.cc:374] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.649118 13244 raft_consensus.cc:729] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.649894 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7984f6e0fe485fa0c15fd13c3f428e (DEFAULT_TABLE table=default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.651983 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc56299ed854d6c8ef201127692fd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.653322 13249 tablet_bootstrap.cc:492] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.653554 13249 ts_tablet_manager.cc:1367] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.127s user 0.046s sys 0.053s
I20240506 19:56:43.652714 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc30f4690b24c5ea8ab2f7a26122992 (DEFAULT_TABLE table=default.Table_6f7629baef0946e6aa92b1cbdef4d362 [id=e57f28ac2f0e4ad49895a608f139c0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.655467 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad7076ed00f4aec9cb92ba7860932b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.656587 13245 tablet_bootstrap.cc:654] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.663046 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet aae313961a914ed485f866d85785a788 (table default.Table_cc5c3f3202074991b8db4134d5824ad4 [id=a63e1c9e058a4501831a67516affe578]). Expected version 1 got 0
I20240506 19:56:43.663277 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
W20240506 19:56:43.665565 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85365758f6a64536ab61e8349de63dea consensus state has no leader
I20240506 19:56:43.666821 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_67154b63102e41e0b35410affe262e62" } modify_external_catalogs: true
I20240506 19:56:43.666950 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45a853e310473d844a423fbefc553a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.667003 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_67154b63102e41e0b35410affe262e62" } modify_external_catalogs: true
I20240506 19:56:43.655753 12781 leader_election.cc:302] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:43.668344 13245 tablet_bootstrap.cc:492] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.668531 13245 ts_tablet_manager.cc:1367] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.002s
I20240506 19:56:43.668942 13244 consensus_queue.cc:260] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.670230 13245 raft_consensus.cc:348] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.670511 13245 raft_consensus.cc:374] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.670585 13245 raft_consensus.cc:729] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.670859 13245 consensus_queue.cc:260] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.671998 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
I20240506 19:56:43.672784 13249 raft_consensus.cc:348] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.673027 13249 raft_consensus.cc:374] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.673100 13249 raft_consensus.cc:729] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.673369 13249 consensus_queue.cc:260] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.673923 13249 ts_tablet_manager.cc:1398] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240506 19:56:43.674243 13249 tablet_bootstrap.cc:492] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.676088 13249 tablet_bootstrap.cc:654] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.677928 13245 ts_tablet_manager.cc:1398] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240506 19:56:43.680789 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7984f6e0fe485fa0c15fd13c3f428e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.682268 13245 tablet_bootstrap.cc:492] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:43.684399 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet d05c5ca7ff864fa98f3494032729e978 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.687172 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 07755e37a20646a3b1a7b900d54a077c with delete_type TABLET_DATA_DELETED (replaced by tablet 034d6309de0740d0a5806b3847e334cb) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:43.690249 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 179348us
I20240506 19:56:43.693497 13842 raft_consensus.cc:2726] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.693498 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6097fe368a894f648b097912e16e361f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.694777 13243 ts_tablet_manager.cc:1888] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.694425 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.695626 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.695402 13243 log.cc:1196] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ea25fb8e42234b82885b8fb2275d4925
I20240506 19:56:43.698246 13243 ts_tablet_manager.cc:1909] T ea25fb8e42234b82885b8fb2275d4925 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:43.698801 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:43.702128 13244 ts_tablet_manager.cc:1398] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.070s user 0.002s sys 0.001s
W20240506 19:56:43.702361 12781 consensus_peers.cc:485] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5725b47e43244f0af881b07e145b3aa. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:43.702589 13244 tablet_bootstrap.cc:492] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:43.704778 12781 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 30d819d1237f4798b4847fdf378cbc62
I20240506 19:56:43.705996 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc2c509a06554b9db6d5e0be97e159db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.712741 12781 leader_election.cc:302] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:43.719105 13842 raft_consensus.cc:2726] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.722038 13244 tablet_bootstrap.cc:654] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.726985 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8a45a853e310473d844a423fbefc553a
I20240506 19:56:43.728411 13246 ts_tablet_manager.cc:1888] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:43.728211 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aded0cd817a0451cbbb77322ae0b30b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.728539 13246 log.cc:1196] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/9c18888ae3d6420ca1de3b13fc36cf6c
I20240506 19:56:43.728926 13246 ts_tablet_manager.cc:1909] T 9c18888ae3d6420ca1de3b13fc36cf6c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:43.729138 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aded0cd817a0451cbbb77322ae0b30b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.730115 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45a853e310473d844a423fbefc553a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.731361 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45a853e310473d844a423fbefc553a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.731829 13245 tablet_bootstrap.cc:654] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.732319 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc30f4690b24c5ea8ab2f7a26122992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.737601 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7984f6e0fe485fa0c15fd13c3f428e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.741395 13353 raft_consensus.cc:482] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.741681 13353 raft_consensus.cc:504] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.742630 13243 tablet_replica.cc:330] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:43.742664 13353 leader_election.cc:288] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:43.742854 13243 raft_consensus.cc:2229] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.743059 13243 raft_consensus.cc:2258] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.743726 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2005ef58fd4fcea16abe771ee385f8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:43.750231 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ea25fb8e42234b82885b8fb2275d4925 (table default.Table_685d0e00c37640ab99bfe74c85c1c0f3 [id=8b3a851d7ee4438daf7440e2af8f9885]) successfully deleted
W20240506 19:56:43.750680 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 64485c3c13e94055a019f2ef7be1f2fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64485c3c13e94055a019f2ef7be1f2fe already in progress: creating tablet
W20240506 19:56:43.750938 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef03fcaa05d04ebd9f262f43ff7c02e4 already in progress: creating tablet
W20240506 19:56:43.751174 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4b4ad903e75547fa829b82cc722fef1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4ad903e75547fa829b82cc722fef1e already in progress: creating tablet
W20240506 19:56:43.751384 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b2a5a5e193f245de96cc82986e292f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a5a5e193f245de96cc82986e292f05 already in progress: creating tablet
W20240506 19:56:43.751612 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 31c440555cfe4195b3ae86e1f1eb6a34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31c440555cfe4195b3ae86e1f1eb6a34 already in progress: creating tablet
I20240506 19:56:43.752710 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 65da8b15e0c6496bb86b47c49e348af0
I20240506 19:56:43.753716 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_503b7b916de7483fa1a71e6e6db2cc48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.754038 13246 tablet_replica.cc:330] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:43.754262 13246 raft_consensus.cc:2229] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:43.754365 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503b7b916de7483fa1a71e6e6db2cc48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.754459 13246 raft_consensus.cc:2258] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.755088 13246 ts_tablet_manager.cc:1875] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.755611 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 65da8b15e0c6496bb86b47c49e348af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.756134 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 452a83470585433e8def7174300d3a7f
I20240506 19:56:43.756873 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7714de27e50a4613bb9ad3a6a31b96aa" } new_table_name: "default.Table_9197190885234701b5933420d98f8ca8" modify_external_catalogs: true
I20240506 19:56:43.758878 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 08dafb661b2443f5afadc6816868bac8
I20240506 19:56:43.758766 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b889456df2234d07a29a7c14cf5270ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.759622 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8937af041a2e4c8faa1bc31f4d1e4b05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.759769 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b889456df2234d07a29a7c14cf5270ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:43.760072 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8937af041a2e4c8faa1bc31f4d1e4b05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.761366 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 452a83470585433e8def7174300d3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.743619 13243 ts_tablet_manager.cc:1875] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:43.744617 12922 leader_election.cc:341] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.763058 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 08dafb661b2443f5afadc6816868bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.763309 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2ff3435c11854649b7daf74f3d8c33d7
I20240506 19:56:43.763868 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 880ecc75bab446e39c687f3c8b740361 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:43.751302 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45a853e310473d844a423fbefc553a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.765664 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_942245e6c2a94a96af9206be9765c407" } new_table_name: "default.Table_fadc9a32c1fb42dc858cd67f8e78e34f" modify_external_catalogs: true
I20240506 19:56:43.766911 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:43.766978 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8448ad8e9384edba7202d0bdce516a4 has no consensus state
I20240506 19:56:43.767338 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 65da8b15e0c6496bb86b47c49e348af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.767338 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e415d116a2364144bd571ef2ea61d225"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.767668 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 452a83470585433e8def7174300d3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:43.767802 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e415d116a2364144bd571ef2ea61d225 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.769524 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]) successfully deleted
I20240506 19:56:43.769800 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:43.769858 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 378759ca310f453fba45d48879e2dd1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 378759ca310f453fba45d48879e2dd1a already in progress: creating tablet
W20240506 19:56:43.770129 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 116df9bd6ab04a30ba09861e765d7267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 116df9bd6ab04a30ba09861e765d7267 already in progress: creating tablet
I20240506 19:56:43.770155 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 08dafb661b2443f5afadc6816868bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:43.770457 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4b4ad903e75547fa829b82cc722fef1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4ad903e75547fa829b82cc722fef1e already in progress: creating tablet
W20240506 19:56:43.770666 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 64485c3c13e94055a019f2ef7be1f2fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64485c3c13e94055a019f2ef7be1f2fe already in progress: creating tablet
W20240506 19:56:43.770865 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef03fcaa05d04ebd9f262f43ff7c02e4 already in progress: creating tablet
W20240506 19:56:43.771085 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b2a5a5e193f245de96cc82986e292f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a5a5e193f245de96cc82986e292f05 already in progress: creating tablet
W20240506 19:56:43.771261 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 31c440555cfe4195b3ae86e1f1eb6a34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31c440555cfe4195b3ae86e1f1eb6a34 already in progress: creating tablet
I20240506 19:56:43.771507 13243 ts_tablet_manager.cc:1888] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.771574 13353 raft_consensus.cc:482] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.771744 13353 raft_consensus.cc:504] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.772420 13353 leader_election.cc:288] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:43.773029 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36b4d51b100486c831a41a52d81fd66" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:43.774008 12922 leader_election.cc:341] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:43.775669 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6449ebda2e1b4b0088a1c4bd083cf123"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.776374 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6449ebda2e1b4b0088a1c4bd083cf123 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.777822 13249 tablet_bootstrap.cc:492] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.777985 13249 ts_tablet_manager.cc:1367] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.054s
I20240506 19:56:43.778383 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 65da8b15e0c6496bb86b47c49e348af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.778825 13249 raft_consensus.cc:348] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.779039 13249 raft_consensus.cc:374] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.779129 13249 raft_consensus.cc:729] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.779868 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.780541 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 08dafb661b2443f5afadc6816868bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.780838 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 452a83470585433e8def7174300d3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.784065 13249 consensus_queue.cc:260] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.784832 13249 ts_tablet_manager.cc:1398] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:43.785245 13249 tablet_bootstrap.cc:492] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.787285 13249 tablet_bootstrap.cc:654] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.789801 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.790484 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cc5c3f3202074991b8db4134d5824ad4" } new_table_name: "default.Table_437b2de81ba04968abe8db40165a9db0" modify_external_catalogs: true
W20240506 19:56:43.788669 12781 consensus_peers.cc:485] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2502b423bd1040539fb1a3ebbece3c1c. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:43.795933 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 78465.4us
I20240506 19:56:43.796952 13246 ts_tablet_manager.cc:1888] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:43.796983 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58c88f80f5b34672b7c3a4b604469bf1 consensus state has no leader
I20240506 19:56:43.797116 13246 log.cc:1196] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a36b4d51b100486c831a41a52d81fd66
I20240506 19:56:43.797479 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c56ef126a4ca48b6a379a6771bc1c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.797708 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 880ecc75bab446e39c687f3c8b740361
I20240506 19:56:43.798346 12522 catalog_manager.cc:6554] ReplaceTablet: tablet 880ecc75bab446e39c687f3c8b740361 deleted and replaced by tablet 7a4a9693c9fb411db07847fede883920
I20240506 19:56:43.798791 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2005ef58fd4fcea16abe771ee385f8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.799278 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36b4d51b100486c831a41a52d81fd66" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:43.800033 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 880ecc75bab446e39c687f3c8b740361 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4a9693c9fb411db07847fede883920) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.771669 13243 log.cc:1196] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:43.800797 13243 ts_tablet_manager.cc:1909] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:43.801470 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 880ecc75bab446e39c687f3c8b740361 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4a9693c9fb411db07847fede883920) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:43.802683 12921 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.802860 12921 leader_election.cc:302] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:43.803081 12921 raft_consensus.cc:2670] T 618c56ef126a4ca48b6a379a6771bc1c P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:43.803404 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 618c56ef126a4ca48b6a379a6771bc1c (table default.Table_b50238125ad14920bf820245406918c3 [id=7f61579ec04443a6992e572f3e52f980]) successfully deleted
I20240506 19:56:43.803697 13246 ts_tablet_manager.cc:1909] T a36b4d51b100486c831a41a52d81fd66 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:43.803702 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 59b022c086d141a19a9b35c4a8c8bef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59b022c086d141a19a9b35c4a8c8bef9 already in progress: creating tablet
W20240506 19:56:43.804047 12921 consensus_peers.cc:485] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:43.804227 12921 leader_election.cc:341] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.804324 12921 leader_election.cc:302] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:43.804770 12921 leader_election.cc:341] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:43.804895 12921 leader_election.cc:302] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:43.807996 13838 raft_consensus.cc:2726] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.808179 13838 raft_consensus.cc:2726] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.812275 13249 tablet_bootstrap.cc:492] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.812515 13249 ts_tablet_manager.cc:1367] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240506 19:56:43.813318 13249 raft_consensus.cc:348] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.813645 13249 raft_consensus.cc:374] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.813735 13249 raft_consensus.cc:729] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.814002 13249 consensus_queue.cc:260] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.814625 13249 ts_tablet_manager.cc:1398] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:43.816484 13249 tablet_bootstrap.cc:492] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:43.818317 13249 tablet_bootstrap.cc:654] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.821234 13245 tablet_bootstrap.cc:492] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:43.821427 13245 ts_tablet_manager.cc:1367] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.140s user 0.001s sys 0.009s
I20240506 19:56:43.821749 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8335f9308046279a25a4a95d8e0f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:43.822788 13245 raft_consensus.cc:348] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.823107 13245 raft_consensus.cc:374] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.823236 13245 raft_consensus.cc:729] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.823530 13245 consensus_queue.cc:260] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:43.824754 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aae313961a914ed485f866d85785a788 consensus state has no leader
W20240506 19:56:43.825670 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee69d8e09c2f4c898ce547ec396ca0e6 consensus state has no leader
I20240506 19:56:43.828581 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 07755e37a20646a3b1a7b900d54a077c with delete_type TABLET_DATA_DELETED (replaced by tablet 034d6309de0740d0a5806b3847e334cb) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:43.828959 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ece0ad4df247eb9d380ed1c755ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.847195 13245 ts_tablet_manager.cc:1398] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240506 19:56:43.848002 13244 tablet_bootstrap.cc:492] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:43.851624 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1b51c240a4b1aad65839679f7bba7 (DEFAULT_TABLE table=default.Table_73585d09b2764941be3d724930ffd097 [id=e0203374d4df488eb5f7b55c13ed9902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.852103 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd1b51c240a4b1aad65839679f7bba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.853192 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 84a6b9f9665945e5ab30c034a26005db (DEFAULT_TABLE table=default.Table_9e3f4bda34084998998a5a34a7717ffd [id=f031e266448b45adbdaaa7a84265da5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.853663 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a6b9f9665945e5ab30c034a26005db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.857734 13244 ts_tablet_manager.cc:1367] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.146s user 0.005s sys 0.000s
I20240506 19:56:43.858721 13242 ts_tablet_manager.cc:1888] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:43.860193 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 880ecc75bab446e39c687f3c8b740361 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4a9693c9fb411db07847fede883920) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.860383 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 7763959136c44674888916321bd6da24 (DEFAULT_TABLE table=default.Table_3612dd9500444861ac651d75a8b106a8 [id=aaa5e060d97a4d12a2632cc487630429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.860590 12955 tablet_service.cc:1467] Processing CreateTablet for tablet cfc7de255d7049f9a69a4193a50cda84 (DEFAULT_TABLE table=default.Table_aeb68c202cc642e6908287b378e38344 [id=d323713c509740a3ad2616eb8fe68337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.860836 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7763959136c44674888916321bd6da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.861507 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc7de255d7049f9a69a4193a50cda84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.861553 13243 tablet_replica.cc:330] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:43.861657 12952 tablet_service.cc:1467] Processing CreateTablet for tablet f68f57002f8144caaaf37c2838f98e1f (DEFAULT_TABLE table=default.Table_768e769dac9046d582bf4c77e1221d9d [id=e2fa2ca551cf45f6ab9b6d22b0c84e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.858914 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 84a6b9f9665945e5ab30c034a26005db (DEFAULT_TABLE table=default.Table_9e3f4bda34084998998a5a34a7717ffd [id=f031e266448b45adbdaaa7a84265da5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.861784 13095 tablet_service.cc:1467] Processing CreateTablet for tablet f68f57002f8144caaaf37c2838f98e1f (DEFAULT_TABLE table=default.Table_768e769dac9046d582bf4c77e1221d9d [id=e2fa2ca551cf45f6ab9b6d22b0c84e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.861968 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 9fce71f5451748b5bda3babdc7d07b86 (DEFAULT_TABLE table=default.Table_9f13287a26d24d28813ef3727d8e6844 [id=ad17c57f0f784939a3f5413480094d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.862059 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68f57002f8144caaaf37c2838f98e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.862129 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a6b9f9665945e5ab30c034a26005db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.862177 13104 tablet_service.cc:1467] Processing CreateTablet for tablet a8448ad8e9384edba7202d0bdce516a4 (DEFAULT_TABLE table=default.Table_f8a1ac708a7d4f20bb111b87031710c9 [id=44207be9fc564b5bba7a4513e0b3c1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.863554 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4379e1f1340668557f7915f1d1453 (DEFAULT_TABLE table=default.Table_a24848c14dfc4d0ca604aabf57ec28d4 [id=99b553da739246f1b28a349d9b0f3462]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.863731 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 9e17d0a261734d45866525b35de7cba3 (DEFAULT_TABLE table=default.Table_a9407b54478a4b51b62ed049b31e30bc [id=9608493a271d4247a4aac7bc3e6d4bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.864059 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4379e1f1340668557f7915f1d1453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.864275 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e17d0a261734d45866525b35de7cba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.864528 12954 tablet_service.cc:1467] Processing CreateTablet for tablet a8448ad8e9384edba7202d0bdce516a4 (DEFAULT_TABLE table=default.Table_f8a1ac708a7d4f20bb111b87031710c9 [id=44207be9fc564b5bba7a4513e0b3c1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.859691 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 9e17d0a261734d45866525b35de7cba3 (DEFAULT_TABLE table=default.Table_a9407b54478a4b51b62ed049b31e30bc [id=9608493a271d4247a4aac7bc3e6d4bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.861050 13092 tablet_service.cc:1467] Processing CreateTablet for tablet cfc7de255d7049f9a69a4193a50cda84 (DEFAULT_TABLE table=default.Table_aeb68c202cc642e6908287b378e38344 [id=d323713c509740a3ad2616eb8fe68337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.861197 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 7763959136c44674888916321bd6da24 (DEFAULT_TABLE table=default.Table_3612dd9500444861ac651d75a8b106a8 [id=aaa5e060d97a4d12a2632cc487630429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.865770 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fce71f5451748b5bda3babdc7d07b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.866015 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7763959136c44674888916321bd6da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.861515 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1b51c240a4b1aad65839679f7bba7 (DEFAULT_TABLE table=default.Table_73585d09b2764941be3d724930ffd097 [id=e0203374d4df488eb5f7b55c13ed9902]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.866334 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8448ad8e9384edba7202d0bdce516a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.866529 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd1b51c240a4b1aad65839679f7bba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.867841 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e17d0a261734d45866525b35de7cba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.861652 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4379e1f1340668557f7915f1d1453 (DEFAULT_TABLE table=default.Table_a24848c14dfc4d0ca604aabf57ec28d4 [id=99b553da739246f1b28a349d9b0f3462]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.869614 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4379e1f1340668557f7915f1d1453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.874111 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68f57002f8144caaaf37c2838f98e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.877574 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc7de255d7049f9a69a4193a50cda84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.877635 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8448ad8e9384edba7202d0bdce516a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.879256 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet a8b3ed0b56aa478abb44347afca8fcf6 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:43.881103 12819 tablet_service.cc:1467] Processing CreateTablet for tablet f68f57002f8144caaaf37c2838f98e1f (DEFAULT_TABLE table=default.Table_768e769dac9046d582bf4c77e1221d9d [id=e2fa2ca551cf45f6ab9b6d22b0c84e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.884443 13242 log.cc:1196] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/581aae0889654903933bef3bf1026954
I20240506 19:56:43.884896 13242 ts_tablet_manager.cc:1909] T 581aae0889654903933bef3bf1026954 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:43.885279 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 388) took 3024 ms. Trace:
I20240506 19:56:43.885412 13242 rpcz_store.cc:276] 0506 19:56:40.860673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.860847 (+ 174us) service_pool.cc:225] Handling call
0506 19:56:43.465010 (+2604163us) ts_tablet_manager.cc:1120] Deleting tablet 581aae0889654903933bef3bf1026954
0506 19:56:43.465025 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:43.643726 (+178701us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:43.836907 (+193181us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:43.885218 (+ 48311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2603901}
I20240506 19:56:43.885785 13242 tablet_replica.cc:330] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:43.885964 13242 raft_consensus.cc:2229] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:43.886178 13242 raft_consensus.cc:2258] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:43.886322 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0921ada34d224bd7b8bc59edc8627549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:43.886355 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_04311b0062d745e3ae935a1abe99343e" } new_table_name: "default.Table_6ed1bd3a98704451b7eb00cfa5de6e6d" modify_external_catalogs: true
I20240506 19:56:43.886758 13242 ts_tablet_manager.cc:1875] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:43.884466 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1b51c240a4b1aad65839679f7bba7 (DEFAULT_TABLE table=default.Table_73585d09b2764941be3d724930ffd097 [id=e0203374d4df488eb5f7b55c13ed9902]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:43.886956 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0921ada34d224bd7b8bc59edc8627549 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.888921 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 7763959136c44674888916321bd6da24 (DEFAULT_TABLE table=default.Table_3612dd9500444861ac651d75a8b106a8 [id=aaa5e060d97a4d12a2632cc487630429]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.891386 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 9e17d0a261734d45866525b35de7cba3 (DEFAULT_TABLE table=default.Table_a9407b54478a4b51b62ed049b31e30bc [id=9608493a271d4247a4aac7bc3e6d4bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.892679 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68f57002f8144caaaf37c2838f98e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.894253 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 581aae0889654903933bef3bf1026954 (table default.Table_ab69ac48aa7a4934a9f383276277cf10 [id=493f88f528114491b898c4a1156712fc]) successfully deleted
I20240506 19:56:43.894603 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 84a6b9f9665945e5ab30c034a26005db (DEFAULT_TABLE table=default.Table_9e3f4bda34084998998a5a34a7717ffd [id=f031e266448b45adbdaaa7a84265da5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.899256 13244 raft_consensus.cc:348] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.900300 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 9fce71f5451748b5bda3babdc7d07b86 (DEFAULT_TABLE table=default.Table_9f13287a26d24d28813ef3727d8e6844 [id=ad17c57f0f784939a3f5413480094d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.898226 12814 tablet_service.cc:1467] Processing CreateTablet for tablet a8448ad8e9384edba7202d0bdce516a4 (DEFAULT_TABLE table=default.Table_f8a1ac708a7d4f20bb111b87031710c9 [id=44207be9fc564b5bba7a4513e0b3c1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.900920 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4379e1f1340668557f7915f1d1453 (DEFAULT_TABLE table=default.Table_a24848c14dfc4d0ca604aabf57ec28d4 [id=99b553da739246f1b28a349d9b0f3462]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.901523 12809 tablet_service.cc:1467] Processing CreateTablet for tablet cfc7de255d7049f9a69a4193a50cda84 (DEFAULT_TABLE table=default.Table_aeb68c202cc642e6908287b378e38344 [id=d323713c509740a3ad2616eb8fe68337]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.902482 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e17d0a261734d45866525b35de7cba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.904377 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 9fce71f5451748b5bda3babdc7d07b86 (DEFAULT_TABLE table=default.Table_9f13287a26d24d28813ef3727d8e6844 [id=ad17c57f0f784939a3f5413480094d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:43.905010 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fce71f5451748b5bda3babdc7d07b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.906479 13244 raft_consensus.cc:374] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.906603 13244 raft_consensus.cc:729] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.907622 13244 consensus_queue.cc:260] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.908196 13244 ts_tablet_manager.cc:1398] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240506 19:56:43.911432 13805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:43.911698 13805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:43.911765 13244 tablet_bootstrap.cc:492] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:43.912307 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a6b9f9665945e5ab30c034a26005db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.913767 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7763959136c44674888916321bd6da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.925525 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4379e1f1340668557f7915f1d1453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.926789 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fce71f5451748b5bda3babdc7d07b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.928669 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8448ad8e9384edba7202d0bdce516a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.933454 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc7de255d7049f9a69a4193a50cda84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.937436 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd1b51c240a4b1aad65839679f7bba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:43.945273 13842 raft_consensus.cc:482] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:43.945497 13842 raft_consensus.cc:504] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.946348 13842 leader_election.cc:288] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:43.947306 12781 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 8f66f1e46cce47639fe644fb4ed2e5c0
W20240506 19:56:43.947448 12783 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:43.947571 12783 leader_election.cc:302] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:43.949081 13842 raft_consensus.cc:2726] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:43.950870 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66f1e46cce47639fe644fb4ed2e5c0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca394b7f53ce445e8778dd25d0cf25c7) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:43.952065 13244 tablet_bootstrap.cc:654] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:43.961817 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b889456df2234d07a29a7c14cf5270ff" } modify_external_catalogs: true
I20240506 19:56:43.962105 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b889456df2234d07a29a7c14cf5270ff" } modify_external_catalogs: true
I20240506 19:56:43.962735 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b40bb86f59744dc387a12a83d5dc8b9b" } modify_external_catalogs: true
I20240506 19:56:43.962908 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b40bb86f59744dc387a12a83d5dc8b9b" } modify_external_catalogs: true
I20240506 19:56:43.973145 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_138cdbf4ae874f409ca3414d505a1217" } modify_external_catalogs: true
I20240506 19:56:43.973434 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_138cdbf4ae874f409ca3414d505a1217" } modify_external_catalogs: true
I20240506 19:56:43.973601 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_826f8c31905e4a76b9bd194c717a5853" } modify_external_catalogs: true
I20240506 19:56:43.973814 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_826f8c31905e4a76b9bd194c717a5853" } modify_external_catalogs: true
I20240506 19:56:43.974112 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b1b710ee5f9941a08a8db2bd7122ef25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.974717 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1b710ee5f9941a08a8db2bd7122ef25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.976029 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cddf9ad090cc4ac690a9a68b8600791e" } modify_external_catalogs: true
I20240506 19:56:43.976194 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cddf9ad090cc4ac690a9a68b8600791e" } modify_external_catalogs: true
I20240506 19:56:43.976742 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_44dc34cdfbc44f1791118c880102c3d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:43.977268 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44dc34cdfbc44f1791118c880102c3d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:43.978683 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_437b2de81ba04968abe8db40165a9db0" } new_table_name: "default.Table_632142c0692a45b69cba3c19adb0b444" modify_external_catalogs: true
I20240506 19:56:43.996307 13249 tablet_bootstrap.cc:492] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:43.996604 13249 ts_tablet_manager.cc:1367] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.180s user 0.035s sys 0.062s
I20240506 19:56:43.997723 13249 raft_consensus.cc:348] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:43.998255 13249 raft_consensus.cc:374] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:43.998386 13249 raft_consensus.cc:729] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:43.998680 13249 consensus_queue.cc:260] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.000489 13249 ts_tablet_manager.cc:1398] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:44.001084 13249 tablet_bootstrap.cc:492] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.003651 13249 tablet_bootstrap.cc:654] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:44.005014 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e22e5a015f74526b49363bc789cb4ed consensus state has no leader
I20240506 19:56:44.008708 13243 raft_consensus.cc:2229] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.008941 13243 raft_consensus.cc:2258] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.009743 13243 ts_tablet_manager.cc:1875] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.016335 13249 tablet_bootstrap.cc:492] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.016633 13249 ts_tablet_manager.cc:1367] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20240506 19:56:44.017493 13245 tablet_bootstrap.cc:492] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:44.017508 13249 raft_consensus.cc:348] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.017869 13249 raft_consensus.cc:374] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.017987 13249 raft_consensus.cc:729] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.018388 13249 consensus_queue.cc:260] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.018770 13246 tablet_replica.cc:330] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:44.018981 13246 raft_consensus.cc:2229] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:44.019279 13246 pending_rounds.cc:62] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:44.019325 13249 ts_tablet_manager.cc:1398] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:44.019359 13246 pending_rounds.cc:68] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744052768329728 op_type: NO_OP noop_request { }
I20240506 19:56:44.019516 13246 raft_consensus.cc:2866] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:44.019634 13246 raft_consensus.cc:2258] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.019971 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a36b4d51b100486c831a41a52d81fd66 (table default.Table_e5d18039b7f04edaa0c985be5260be35 [id=7a6f6d0d394340ea93a13720ea7b769e]) successfully deleted
I20240506 19:56:44.020304 13249 tablet_bootstrap.cc:492] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.025375 13244 tablet_bootstrap.cc:492] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:44.025590 13244 ts_tablet_manager.cc:1367] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.117s user 0.006s sys 0.002s
I20240506 19:56:44.027712 13244 raft_consensus.cc:348] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:44.027966 13244 raft_consensus.cc:374] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.028008 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet b5a443c0d35741b9966b32a7808a97e2 (table default.Table_1a5485067c1a4e529059f5434babbdc7 [id=54286945a7554349b48aa6511f84326c]). Expected version 1 got 0
I20240506 19:56:44.028111 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1513820768504912ad5f7b9966ada614"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.028688 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1513820768504912ad5f7b9966ada614 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.028877 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b1f83bed002d4d3fbf1a4b5b95f0ea54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:44.029083 13244 raft_consensus.cc:729] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:44.029475 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1f83bed002d4d3fbf1a4b5b95f0ea54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.029417 13244 consensus_queue.cc:260] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:44.029908 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_491916e47bc640f199cb29f14c922469"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.030452 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_491916e47bc640f199cb29f14c922469 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.032200 13245 tablet_bootstrap.cc:654] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.032151 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2b9776133219459d8dfd0ea02428d80f" } new_table_name: "default.Table_41de97bb590e4b7ebb026aad76af73b4" modify_external_catalogs: true
W20240506 19:56:44.033567 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aae313961a914ed485f866d85785a788 consensus state has no leader
I20240506 19:56:44.033993 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c602a53461334dcca2cdcebd0af292cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.034533 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c602a53461334dcca2cdcebd0af292cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.042652 13243 ts_tablet_manager.cc:1888] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.042836 13243 log.cc:1196] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a36b4d51b100486c831a41a52d81fd66
I20240506 19:56:44.043248 13243 ts_tablet_manager.cc:1909] T a36b4d51b100486c831a41a52d81fd66 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.043777 12962 tablet_service.cc:1467] Processing CreateTablet for tablet bb04d06a2b084079ac6357415b124752 (DEFAULT_TABLE table=default.Table_8937af041a2e4c8faa1bc31f4d1e4b05 [id=3b33c8987c924f3aa02ad543f46a1530]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.044018 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 812f51b7d2044c3ba670c90f70a32c63 (DEFAULT_TABLE table=default.Table_fcad4ec094e649709cb1cf84577e7483 [id=3bfc6a7cd8d940b298a8bd52f65e78d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.044222 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb04d06a2b084079ac6357415b124752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.044484 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812f51b7d2044c3ba670c90f70a32c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.045079 12954 tablet_service.cc:1467] Processing CreateTablet for tablet ee62b918bba543a6b3e467fd8dc59878 (DEFAULT_TABLE table=default.Table_b889456df2234d07a29a7c14cf5270ff [id=4e145d2c76034b32bf6ef14089a0e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.045603 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 56de924d3f104a7da4f4c37924875555 (DEFAULT_TABLE table=default.Table_6449ebda2e1b4b0088a1c4bd083cf123 [id=2126bfc39a3b4b34b6edf5460144fb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.045673 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee62b918bba543a6b3e467fd8dc59878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.046162 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56de924d3f104a7da4f4c37924875555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.050068 12955 tablet_service.cc:1467] Processing CreateTablet for tablet a207f4cfc01943ff9059f025925572c5 (DEFAULT_TABLE table=default.Table_0f727575df874202b2cdd6ca62d5727f [id=6299eacabfd0428a81fcd18ce665cec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.050712 13094 tablet_service.cc:1467] Processing CreateTablet for tablet ee62b918bba543a6b3e467fd8dc59878 (DEFAULT_TABLE table=default.Table_b889456df2234d07a29a7c14cf5270ff [id=4e145d2c76034b32bf6ef14089a0e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.051152 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee62b918bba543a6b3e467fd8dc59878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.052736 12952 tablet_service.cc:1467] Processing CreateTablet for tablet f55586b153b44c31a298a8d46bd8c532 (DEFAULT_TABLE table=default.Table_538bc8e0e1604e09bed86b488cf7ada0 [id=73594db848b740e688f6bc386e5fd717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.053596 12963 tablet_service.cc:1467] Processing CreateTablet for tablet c0c7cae21c714148a37cab389d116e91 (DEFAULT_TABLE table=default.Table_aded0cd817a0451cbbb77322ae0b30b0 [id=980453fcdcee42e087de3ce4065db7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.055697 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4a9693c9fb411db07847fede883920 (DEFAULT_TABLE table=default.Table_a6c3a4c9339a480db1e49222fe7e0368 [id=0b3a486ff6d24c8795de3e53e83021a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.056206 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4a9693c9fb411db07847fede883920 (DEFAULT_TABLE table=default.Table_a6c3a4c9339a480db1e49222fe7e0368 [id=0b3a486ff6d24c8795de3e53e83021a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.056795 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 56de924d3f104a7da4f4c37924875555 (DEFAULT_TABLE table=default.Table_6449ebda2e1b4b0088a1c4bd083cf123 [id=2126bfc39a3b4b34b6edf5460144fb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.057178 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56de924d3f104a7da4f4c37924875555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.057394 13249 tablet_bootstrap.cc:654] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.058912 13097 tablet_service.cc:1467] Processing CreateTablet for tablet f55586b153b44c31a298a8d46bd8c532 (DEFAULT_TABLE table=default.Table_538bc8e0e1604e09bed86b488cf7ada0 [id=73594db848b740e688f6bc386e5fd717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.059402 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55586b153b44c31a298a8d46bd8c532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.059408 12817 tablet_service.cc:1467] Processing CreateTablet for tablet b5809556edb740dfb542aae70194e631 (DEFAULT_TABLE table=default.Table_e415d116a2364144bd571ef2ea61d225 [id=154b16b0c6b94668a30c453782899633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.060072 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7715dad8184493b085d891428ad44c (DEFAULT_TABLE table=default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.060211 12822 tablet_service.cc:1467] Processing CreateTablet for tablet c0c7cae21c714148a37cab389d116e91 (DEFAULT_TABLE table=default.Table_aded0cd817a0451cbbb77322ae0b30b0 [id=980453fcdcee42e087de3ce4065db7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.060866 12950 tablet_service.cc:1467] Processing CreateTablet for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 (DEFAULT_TABLE table=default.Table_7881939105214872a57c956985a14360 [id=c91b5a199c00499186aabe5137b4f59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.065781 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4a9693c9fb411db07847fede883920 (DEFAULT_TABLE table=default.Table_a6c3a4c9339a480db1e49222fe7e0368 [id=0b3a486ff6d24c8795de3e53e83021a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.066372 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a9693c9fb411db07847fede883920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.067487 12821 tablet_service.cc:1467] Processing CreateTablet for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 (DEFAULT_TABLE table=default.Table_7881939105214872a57c956985a14360 [id=c91b5a199c00499186aabe5137b4f59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.068189 12949 tablet_service.cc:1467] Processing CreateTablet for tablet b5809556edb740dfb542aae70194e631 (DEFAULT_TABLE table=default.Table_e415d116a2364144bd571ef2ea61d225 [id=154b16b0c6b94668a30c453782899633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.069248 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a207f4cfc01943ff9059f025925572c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.069612 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7715dad8184493b085d891428ad44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.069887 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 812f51b7d2044c3ba670c90f70a32c63 (DEFAULT_TABLE table=default.Table_fcad4ec094e649709cb1cf84577e7483 [id=3bfc6a7cd8d940b298a8bd52f65e78d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.070315 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812f51b7d2044c3ba670c90f70a32c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.073747 13096 tablet_service.cc:1467] Processing CreateTablet for tablet a207f4cfc01943ff9059f025925572c5 (DEFAULT_TABLE table=default.Table_0f727575df874202b2cdd6ca62d5727f [id=6299eacabfd0428a81fcd18ce665cec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.074245 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a207f4cfc01943ff9059f025925572c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.080137 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.081753 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 812f51b7d2044c3ba670c90f70a32c63 (DEFAULT_TABLE table=default.Table_fcad4ec094e649709cb1cf84577e7483 [id=3bfc6a7cd8d940b298a8bd52f65e78d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.082295 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812f51b7d2044c3ba670c90f70a32c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.086504 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c7cae21c714148a37cab389d116e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.089794 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55586b153b44c31a298a8d46bd8c532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.098310 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a9693c9fb411db07847fede883920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.101529 13092 tablet_service.cc:1467] Processing CreateTablet for tablet b5809556edb740dfb542aae70194e631 (DEFAULT_TABLE table=default.Table_e415d116a2364144bd571ef2ea61d225 [id=154b16b0c6b94668a30c453782899633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.103188 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5809556edb740dfb542aae70194e631. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:44.104046 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1e22e5a015f74526b49363bc789cb4ed (table default.Table_6ed1bd3a98704451b7eb00cfa5de6e6d [id=fe54314cddeb4a78bc0d6dcf9ee04d75], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1e22e5a015f74526b49363bc789cb4ed consensus state has no leader
I20240506 19:56:44.111167 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 986f9c83b04f41b49e9b78fce731a90f
I20240506 19:56:44.112154 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 986f9c83b04f41b49e9b78fce731a90f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.114547 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_20af0dc96de14fdf9a880466506fde22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.115237 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20af0dc96de14fdf9a880466506fde22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.115237 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet b5a443c0d35741b9966b32a7808a97e2 (table default.Table_1a5485067c1a4e529059f5434babbdc7 [id=54286945a7554349b48aa6511f84326c]). Expected version 1 got 0
I20240506 19:56:44.115777 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1e8cbfa77d85462baa8b75f47a5c76bf" } new_table_name: "default.Table_456c37fe3dda48289f407577c6618ba9" modify_external_catalogs: true
I20240506 19:56:44.116425 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ef6c459b3eed45d08d709ab6f797aa4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.116794 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc2c509a06554b9db6d5e0be97e159db consensus state has no leader
W20240506 19:56:44.117026 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef6c459b3eed45d08d709ab6f797aa4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.117169 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ba8c2d366b18431e967a2fcdf655e29e" } new_table_name: "default.Table_438b764bbb844c7cbc052c23c33795ec" modify_external_catalogs: true
I20240506 19:56:44.119199 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 40b9731a473449b19a9e95653a9d0830
I20240506 19:56:44.119814 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet d9443cdbaa3c40299b66d216673c4b1a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.120548 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c652feb405ef4f1cab8ee602e9d3eb18
I20240506 19:56:44.121214 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d19db9cff74843e087fd82db2475c29c
I20240506 19:56:44.123263 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_53cda80d706841ef876c3a0a4b51271a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.123829 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53cda80d706841ef876c3a0a4b51271a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.126946 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet d19db9cff74843e087fd82db2475c29c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.127229 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet c652feb405ef4f1cab8ee602e9d3eb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.127980 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ee62b918bba543a6b3e467fd8dc59878
I20240506 19:56:44.127976 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b9731a473449b19a9e95653a9d0830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.128268 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet b529bb9362924caca399491e7d9e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.130633 13245 tablet_bootstrap.cc:492] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:44.130528 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7715dad8184493b085d891428ad44c (DEFAULT_TABLE table=default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.129969 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet ee62b918bba543a6b3e467fd8dc59878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.131539 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5809556edb740dfb542aae70194e631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.132123 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 986f9c83b04f41b49e9b78fce731a90f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.132426 13089 tablet_service.cc:1467] Processing CreateTablet for tablet bb04d06a2b084079ac6357415b124752 (DEFAULT_TABLE table=default.Table_8937af041a2e4c8faa1bc31f4d1e4b05 [id=3b33c8987c924f3aa02ad543f46a1530]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.132886 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet c652feb405ef4f1cab8ee602e9d3eb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.133041 13093 tablet_service.cc:1467] Processing CreateTablet for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 (DEFAULT_TABLE table=default.Table_7881939105214872a57c956985a14360 [id=c91b5a199c00499186aabe5137b4f59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.133448 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet d19db9cff74843e087fd82db2475c29c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.133641 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet ee62b918bba543a6b3e467fd8dc59878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.133777 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7715dad8184493b085d891428ad44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.134307 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a9693c9fb411db07847fede883920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.134752 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c7cae21c714148a37cab389d116e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.134829 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7715dad8184493b085d891428ad44c (DEFAULT_TABLE table=default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.135255 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7715dad8184493b085d891428ad44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.135905 12812 tablet_service.cc:1467] Processing CreateTablet for tablet a207f4cfc01943ff9059f025925572c5 (DEFAULT_TABLE table=default.Table_0f727575df874202b2cdd6ca62d5727f [id=6299eacabfd0428a81fcd18ce665cec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.136631 12818 tablet_service.cc:1467] Processing CreateTablet for tablet bb04d06a2b084079ac6357415b124752 (DEFAULT_TABLE table=default.Table_8937af041a2e4c8faa1bc31f4d1e4b05 [id=3b33c8987c924f3aa02ad543f46a1530]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.137424 12824 tablet_service.cc:1467] Processing CreateTablet for tablet f55586b153b44c31a298a8d46bd8c532 (DEFAULT_TABLE table=default.Table_538bc8e0e1604e09bed86b488cf7ada0 [id=73594db848b740e688f6bc386e5fd717]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.148547 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.149287 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet b38453bf678748399339806d12993c88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.152823 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 986f9c83b04f41b49e9b78fce731a90f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.153394 12809 tablet_service.cc:1467] Processing CreateTablet for tablet ee62b918bba543a6b3e467fd8dc59878 (DEFAULT_TABLE table=default.Table_b889456df2234d07a29a7c14cf5270ff [id=4e145d2c76034b32bf6ef14089a0e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.153836 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb04d06a2b084079ac6357415b124752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.154628 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet d19db9cff74843e087fd82db2475c29c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.154686 13244 ts_tablet_manager.cc:1398] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.045s user 0.003s sys 0.000s
I20240506 19:56:44.154983 13243 tablet_replica.cc:330] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.155189 13243 raft_consensus.cc:2229] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.155390 13243 raft_consensus.cc:2258] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.155988 13243 ts_tablet_manager.cc:1875] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.157791 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a36b4d51b100486c831a41a52d81fd66 (table default.Table_e5d18039b7f04edaa0c985be5260be35 [id=7a6f6d0d394340ea93a13720ea7b769e]) successfully deleted
I20240506 19:56:44.159425 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 56de924d3f104a7da4f4c37924875555 (DEFAULT_TABLE table=default.Table_6449ebda2e1b4b0088a1c4bd083cf123 [id=2126bfc39a3b4b34b6edf5460144fb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.160133 13249 tablet_bootstrap.cc:492] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.160377 13249 ts_tablet_manager.cc:1367] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.140s user 0.036s sys 0.041s
I20240506 19:56:44.161206 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet a8b3ed0b56aa478abb44347afca8fcf6 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.161634 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a207f4cfc01943ff9059f025925572c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.162833 13249 raft_consensus.cc:348] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.162986 13244 tablet_bootstrap.cc:492] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:44.163086 13249 raft_consensus.cc:374] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.163187 13249 raft_consensus.cc:729] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.163429 13249 consensus_queue.cc:260] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.166563 13245 ts_tablet_manager.cc:1367] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.127s user 0.001s sys 0.002s
I20240506 19:56:44.167011 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.166980 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b9731a473449b19a9e95653a9d0830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.167691 13104 tablet_service.cc:1467] Processing CreateTablet for tablet c0c7cae21c714148a37cab389d116e91 (DEFAULT_TABLE table=default.Table_aded0cd817a0451cbbb77322ae0b30b0 [id=980453fcdcee42e087de3ce4065db7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.168143 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c7cae21c714148a37cab389d116e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.169284 13249 ts_tablet_manager.cc:1398] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:44.169693 13249 tablet_bootstrap.cc:492] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.169903 13245 raft_consensus.cc:348] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.170137 13245 raft_consensus.cc:374] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.170221 13245 raft_consensus.cc:729] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.170516 13245 consensus_queue.cc:260] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.171481 13249 tablet_bootstrap.cc:654] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.172407 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.172950 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b9731a473449b19a9e95653a9d0830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.176970 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet c652feb405ef4f1cab8ee602e9d3eb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.183137 13243 ts_tablet_manager.cc:1888] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.183322 13243 log.cc:1196] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:44.183732 13243 ts_tablet_manager.cc:1909] T 07755e37a20646a3b1a7b900d54a077c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.184171 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5809556edb740dfb542aae70194e631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.187187 13243 tablet_replica.cc:330] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.187395 13243 raft_consensus.cc:2229] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.187558 13243 raft_consensus.cc:2258] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.188058 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee62b918bba543a6b3e467fd8dc59878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.188127 13243 ts_tablet_manager.cc:1875] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.173492 13848 raft_consensus.cc:482] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.188725 13848 raft_consensus.cc:504] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.189438 13848 leader_election.cc:288] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.191982 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2519c03738864f6ab5dcfab7bc3fc8aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:44.192211 12986 raft_consensus.cc:2454] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:44.192544 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55586b153b44c31a298a8d46bd8c532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.192837 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56de924d3f104a7da4f4c37924875555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.192858 13057 leader_election.cc:302] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.193490 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2519c03738864f6ab5dcfab7bc3fc8aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.194043 13058 leader_election.cc:341] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.194509 13848 raft_consensus.cc:2781] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:44.194599 13848 raft_consensus.cc:482] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:44.194701 13848 raft_consensus.cc:3037] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.200052 13864 raft_consensus.cc:482] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.200278 13864 raft_consensus.cc:504] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.200512 13848 raft_consensus.cc:504] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.201341 13864 leader_election.cc:288] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.202551 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2519c03738864f6ab5dcfab7bc3fc8aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:44.202558 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 07755e37a20646a3b1a7b900d54a077c (table default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]) successfully deleted
I20240506 19:56:44.180377 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb04d06a2b084079ac6357415b124752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.202790 12986 raft_consensus.cc:3037] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.202890 13243 ts_tablet_manager.cc:1888] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.203048 13243 log.cc:1196] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/0e2005ef58fd4fcea16abe771ee385f8
I20240506 19:56:44.203440 13243 ts_tablet_manager.cc:1909] T 0e2005ef58fd4fcea16abe771ee385f8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.205852 12986 raft_consensus.cc:2454] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:44.206307 13057 leader_election.cc:302] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.206266 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 0e2005ef58fd4fcea16abe771ee385f8 (table default.Table_577b201c2bf345bdbaeb7bda56b8296c [id=0772215de50d467585bec1f234187f97]) successfully deleted
W20240506 19:56:44.206653 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8a45a853e310473d844a423fbefc553a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a45a853e310473d844a423fbefc553a already in progress: creating tablet
W20240506 19:56:44.206889 12922 leader_election.cc:341] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b529bb9362924caca399491e7d9e03cd
W20240506 19:56:44.206926 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 65da8b15e0c6496bb86b47c49e348af0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65da8b15e0c6496bb86b47c49e348af0 already in progress: creating tablet
I20240506 19:56:44.207350 13848 leader_election.cc:288] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.207609 13848 raft_consensus.cc:2781] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:44.207782 13848 raft_consensus.cc:686] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:44.208086 13848 consensus_queue.cc:237] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.209282 13243 tablet_replica.cc:330] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.209380 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.209650 13243 raft_consensus.cc:2229] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.209932 13243 raft_consensus.cc:2258] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.211005 13243 ts_tablet_manager.cc:1875] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.211462 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_53b38bd9c00f46f2abac3c4ca498fb0a" } new_table_name: "default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb" modify_external_catalogs: true
W20240506 19:56:44.215085 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2ff3435c11854649b7daf74f3d8c33d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ff3435c11854649b7daf74f3d8c33d7 already in progress: creating tablet
W20240506 19:56:44.215399 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 08dafb661b2443f5afadc6816868bac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08dafb661b2443f5afadc6816868bac8 already in progress: creating tablet
W20240506 19:56:44.215678 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 452a83470585433e8def7174300d3a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452a83470585433e8def7174300d3a7f already in progress: creating tablet
I20240506 19:56:44.219156 13243 ts_tablet_manager.cc:1888] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.219383 13243 log.cc:1196] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2519c03738864f6ab5dcfab7bc3fc8aa
I20240506 19:56:44.219983 13243 ts_tablet_manager.cc:1909] T 2519c03738864f6ab5dcfab7bc3fc8aa P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.221555 13243 tablet_replica.cc:330] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.221616 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2519c03738864f6ab5dcfab7bc3fc8aa (table default.Table_23d6ef6eb15d489e813fdd5562e5be63 [id=5d61a15249c74e509342122448b76500]) successfully deleted
I20240506 19:56:44.221848 13243 raft_consensus.cc:2229] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.222084 13243 raft_consensus.cc:2258] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:44.222102 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 880ecc75bab446e39c687f3c8b740361 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 880ecc75bab446e39c687f3c8b740361 already in progress: creating tablet
I20240506 19:56:44.223188 13243 ts_tablet_manager.cc:1875] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.223803 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b529bb9362924caca399491e7d9e03cd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.224370 12921 leader_election.cc:341] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.224550 12921 leader_election.cc:302] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.225134 13864 raft_consensus.cc:2726] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:44.224985 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2519c03738864f6ab5dcfab7bc3fc8aa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:44.226949 13058 leader_election.cc:341] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.229207 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.252:36867
--webserver_interface=127.12.48.252
--webserver_port=37261
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 19:56:44.229363 13246 ts_tablet_manager.cc:1875] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.230890 13243 ts_tablet_manager.cc:1888] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.231040 13243 log.cc:1196] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:44.231405 13243 ts_tablet_manager.cc:1909] T cc8335f9308046279a25a4a95d8e0f65 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.232487 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet cc8335f9308046279a25a4a95d8e0f65 (table default.Table_df1733520c4a4ae6a99a78f194d6cf06 [id=64a093df5c7143668da7d3e91eccba2a]) successfully deleted
W20240506 19:56:44.233237 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 986f9c83b04f41b49e9b78fce731a90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 986f9c83b04f41b49e9b78fce731a90f already in progress: creating tablet
W20240506 19:56:44.233846 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d19db9cff74843e087fd82db2475c29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d19db9cff74843e087fd82db2475c29c already in progress: creating tablet
W20240506 19:56:44.234478 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c652feb405ef4f1cab8ee602e9d3eb18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c652feb405ef4f1cab8ee602e9d3eb18 already in progress: creating tablet
W20240506 19:56:44.235009 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 40b9731a473449b19a9e95653a9d0830 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40b9731a473449b19a9e95653a9d0830 already in progress: creating tablet
I20240506 19:56:44.235347 13243 tablet_replica.cc:330] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.235579 13243 raft_consensus.cc:2229] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.235797 13243 raft_consensus.cc:2258] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.236635 13243 ts_tablet_manager.cc:1875] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.236872 13246 ts_tablet_manager.cc:1888] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:44.243299 13243 ts_tablet_manager.cc:1888] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.243541 13243 log.cc:1196] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b529bb9362924caca399491e7d9e03cd
I20240506 19:56:44.244017 13243 ts_tablet_manager.cc:1909] T b529bb9362924caca399491e7d9e03cd P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.245313 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b529bb9362924caca399491e7d9e03cd (table default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c [id=fd3051399ebf4aa2b31bbd709b120d48]) successfully deleted
W20240506 19:56:44.246160 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ee62b918bba543a6b3e467fd8dc59878 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee62b918bba543a6b3e467fd8dc59878 already in progress: creating tablet
I20240506 19:56:44.246585 13243 tablet_replica.cc:330] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.246845 13243 raft_consensus.cc:2229] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.247670 13243 raft_consensus.cc:2258] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.248621 13243 ts_tablet_manager.cc:1875] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.256049 13249 tablet_bootstrap.cc:492] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.256362 13249 ts_tablet_manager.cc:1367] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.014s
I20240506 19:56:44.256678 13246 log.cc:1196] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ea25fb8e42234b82885b8fb2275d4925
I20240506 19:56:44.257409 13246 ts_tablet_manager.cc:1909] T ea25fb8e42234b82885b8fb2275d4925 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:44.257710 13249 raft_consensus.cc:348] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.258123 13249 raft_consensus.cc:374] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.258227 13249 raft_consensus.cc:729] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.258579 13249 consensus_queue.cc:260] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.259281 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 523) took 1056 ms. Trace:
W20240506 19:56:44.259819 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9df3ab9dff5148f9bb335a7209b29bbc consensus state has no leader
W20240506 19:56:44.260466 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 consensus state has no leader
I20240506 19:56:44.264077 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet d05c5ca7ff864fa98f3494032729e978 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.247431 13877 raft_consensus.cc:482] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.254125 13875 raft_consensus.cc:482] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.264950 13877 raft_consensus.cc:504] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.265506 13875 raft_consensus.cc:504] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.259392 13246 rpcz_store.cc:276] 0506 19:56:43.202584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.213413 (+ 10829us) service_pool.cc:225] Handling call
0506 19:56:44.018665 (+805252us) ts_tablet_manager.cc:1120] Deleting tablet ea25fb8e42234b82885b8fb2275d4925
0506 19:56:44.018677 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.233043 (+214366us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.236739 (+ 3696us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.259220 (+ 22481us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":804653}
I20240506 19:56:44.266500 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ea25fb8e42234b82885b8fb2275d4925 (table default.Table_685d0e00c37640ab99bfe74c85c1c0f3 [id=8b3a851d7ee4438daf7440e2af8f9885]) successfully deleted
I20240506 19:56:44.266584 13875 leader_election.cc:288] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.267011 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 525) took 1013 ms. Trace:
I20240506 19:56:44.267278 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05c5ca7ff864fa98f3494032729e978" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:44.267529 12988 raft_consensus.cc:2454] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:44.267457 13057 leader_election.cc:341] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:44.267411 13246 rpcz_store.cc:276] 0506 19:56:43.253934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.258013 (+ 4079us) service_pool.cc:225] Handling call
0506 19:56:44.266564 (+1008551us) ts_tablet_manager.cc:1120] Deleting tablet 59b022c086d141a19a9b35c4a8c8bef9
0506 19:56:44.266582 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.266959 (+ 377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008190}
I20240506 19:56:44.267982 13057 leader_election.cc:302] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.268402 13864 raft_consensus.cc:482] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.268551 13246 tablet_replica.cc:330] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:44.268536 13864 raft_consensus.cc:504] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.268656 13875 raft_consensus.cc:2781] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:44.268750 13875 raft_consensus.cc:482] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:44.268780 13246 raft_consensus.cc:2229] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:44.268847 13875 raft_consensus.cc:3037] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.269059 13246 pending_rounds.cc:62] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:44.269127 13246 pending_rounds.cc:68] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744055639531520 op_type: NO_OP noop_request { }
I20240506 19:56:44.269162 13864 leader_election.cc:288] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.269904 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:44.269935 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:44.270166 13123 raft_consensus.cc:2454] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:44.270509 12921 leader_election.cc:341] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.269266 13246 raft_consensus.cc:2866] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:44.270938 12922 leader_election.cc:302] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.271020 13246 raft_consensus.cc:2258] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.271333 13864 raft_consensus.cc:2781] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:44.271456 13864 raft_consensus.cc:482] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:44.271555 13864 raft_consensus.cc:3037] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.272435 13875 raft_consensus.cc:504] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:44.273695 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 59b022c086d141a19a9b35c4a8c8bef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59b022c086d141a19a9b35c4a8c8bef9 already in progress: creating tablet
I20240506 19:56:44.274296 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05c5ca7ff864fa98f3494032729e978" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:44.274516 12988 raft_consensus.cc:3037] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.254384 13245 ts_tablet_manager.cc:1398] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.088s user 0.008s sys 0.020s
I20240506 19:56:44.275570 13245 tablet_bootstrap.cc:492] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:44.275521 13864 raft_consensus.cc:504] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.276650 13864 leader_election.cc:288] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.278282 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:44.267511 13877 leader_election.cc:288] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:44.282537 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67e8b8be953a4a5784f2d098fd6364e3 consensus state has no leader
I20240506 19:56:44.280715 12988 raft_consensus.cc:2454] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:44.286597 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a69b09047c6a479789e134aaad2b53ad" } new_table_name: "default.Table_8afdb494805e4ba7838632b12912ec36" modify_external_catalogs: true
I20240506 19:56:44.286713 13087 tablet_service.cc:1467] Processing CreateTablet for tablet e262c4458fe5415fac50309294312ca6 (DEFAULT_TABLE table=default.Table_44dc34cdfbc44f1791118c880102c3d5 [id=9f0efd77ee3c48408f0b33c81a2dd76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.287207 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 9be3b844a9334cfabea3946844ae1af8 (DEFAULT_TABLE table=default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.287585 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 567cb184adf14b78aaf12d0bbdcf7826 (DEFAULT_TABLE table=default.Table_0921ada34d224bd7b8bc59edc8627549 [id=14b3734b28354d60b48fa59014220a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.287760 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_19ad839886424d5baf4207f64dbfbd8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.288383 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19ad839886424d5baf4207f64dbfbd8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.285398 13123 raft_consensus.cc:3037] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.291653 13123 raft_consensus.cc:2454] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:44.292181 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d7e3f13ea1c1428b9ca232be80f34d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.292774 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7e3f13ea1c1428b9ca232be80f34d5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.295598 13244 tablet_bootstrap.cc:654] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.295619 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e0368ab9f3a24d2ea6b743689cb3a9b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.296245 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0368ab9f3a24d2ea6b743689cb3a9b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.297390 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05c5ca7ff864fa98f3494032729e978" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:44.286027 13057 leader_election.cc:302] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.298354 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 567cb184adf14b78aaf12d0bbdcf7826 (DEFAULT_TABLE table=default.Table_0921ada34d224bd7b8bc59edc8627549 [id=14b3734b28354d60b48fa59014220a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.298908 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05c5ca7ff864fa98f3494032729e978" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.299690 13058 leader_election.cc:341] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:44.300005 13058 leader_election.cc:341] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.300005 13877 raft_consensus.cc:2781] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:44.281113 13249 ts_tablet_manager.cc:1398] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240506 19:56:44.300653 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07755e37a20646a3b1a7b900d54a077c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.301205 13058 leader_election.cc:341] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.301398 13058 leader_election.cc:302] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.301206 13877 raft_consensus.cc:686] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:44.301947 13877 consensus_queue.cc:237] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.302853 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:44.304145 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567cb184adf14b78aaf12d0bbdcf7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.308768 13875 leader_election.cc:288] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.314226 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 9be3b844a9334cfabea3946844ae1af8 (DEFAULT_TABLE table=default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.314720 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be3b844a9334cfabea3946844ae1af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.316172 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be3b844a9334cfabea3946844ae1af8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:44.317325 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45da091c44824bd6bb8d390a5fb01ef8 consensus state has no leader
I20240506 19:56:44.318639 13245 tablet_bootstrap.cc:654] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.320557 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 567cb184adf14b78aaf12d0bbdcf7826 (DEFAULT_TABLE table=default.Table_0921ada34d224bd7b8bc59edc8627549 [id=14b3734b28354d60b48fa59014220a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.320890 12922 leader_election.cc:302] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.321354 13864 raft_consensus.cc:2781] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:44.321511 13864 raft_consensus.cc:686] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:44.325415 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cc43c588b4804f9aa2bbadd0d65ce064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.326212 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc43c588b4804f9aa2bbadd0d65ce064 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.334247 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_291e179ddb654a8e90d5e5ea4db36e60" } modify_external_catalogs: true
I20240506 19:56:44.334512 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_291e179ddb654a8e90d5e5ea4db36e60" } modify_external_catalogs: true
I20240506 19:56:44.336077 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_268439c35972405584472829b94a5011" } new_table_name: "default.Table_1707b02f8b93477eaa71e7e6e47da51c" modify_external_catalogs: true
I20240506 19:56:44.336225 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_799d22eaba644b6ea3acc774f66b8ba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.336903 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_799d22eaba644b6ea3acc774f66b8ba2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.340885 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1a5485067c1a4e529059f5434babbdc7" } modify_external_catalogs: true
I20240506 19:56:44.341213 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1a5485067c1a4e529059f5434babbdc7" } modify_external_catalogs: true
I20240506 19:56:44.344224 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e262c4458fe5415fac50309294312ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.344477 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7511af3d69ea4e9d921a500168622d24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.345081 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7511af3d69ea4e9d921a500168622d24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.345973 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6ed1bd3a98704451b7eb00cfa5de6e6d" } modify_external_catalogs: true
I20240506 19:56:44.346231 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6ed1bd3a98704451b7eb00cfa5de6e6d" } modify_external_catalogs: true
I20240506 19:56:44.346809 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567cb184adf14b78aaf12d0bbdcf7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.347324 13875 raft_consensus.cc:2726] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:44.347726 13249 tablet_bootstrap.cc:492] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.347851 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567cb184adf14b78aaf12d0bbdcf7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.349285 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 9be3b844a9334cfabea3946844ae1af8 (DEFAULT_TABLE table=default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.321825 13864 consensus_queue.cc:237] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:44.347057 12921 leader_election.cc:341] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.350497 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be3b844a9334cfabea3946844ae1af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.352190 12821 tablet_service.cc:1467] Processing CreateTablet for tablet e262c4458fe5415fac50309294312ca6 (DEFAULT_TABLE table=default.Table_44dc34cdfbc44f1791118c880102c3d5 [id=9f0efd77ee3c48408f0b33c81a2dd76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.353332 12958 tablet_service.cc:1467] Processing CreateTablet for tablet e262c4458fe5415fac50309294312ca6 (DEFAULT_TABLE table=default.Table_44dc34cdfbc44f1791118c880102c3d5 [id=9f0efd77ee3c48408f0b33c81a2dd76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.353717 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e262c4458fe5415fac50309294312ca6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:44.356884 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 55673.4us
W20240506 19:56:44.357182 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 53767.3us
I20240506 19:56:44.360267 13249 tablet_bootstrap.cc:654] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.358892 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet ee62b918bba543a6b3e467fd8dc59878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.366546 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_70f25b0c770549f5b47f5ac2d70cd33c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.367259 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70f25b0c770549f5b47f5ac2d70cd33c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.371356 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bef80a208c25432493da6ad772c8bf32" } modify_external_catalogs: true
I20240506 19:56:44.371573 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bef80a208c25432493da6ad772c8bf32" } modify_external_catalogs: true
I20240506 19:56:44.377640 13242 ts_tablet_manager.cc:1888] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.379009 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e262c4458fe5415fac50309294312ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.385624 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet acd3fc62e9614cf3bde0202ee6832bb3
I20240506 19:56:44.387269 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet acd3fc62e9614cf3bde0202ee6832bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.387310 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet acd3fc62e9614cf3bde0202ee6832bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:44.387758 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da06187c83b94d03ae4ebb67a2873cf0 consensus state has no leader
I20240506 19:56:44.388559 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet acd3fc62e9614cf3bde0202ee6832bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.392086 13842 raft_consensus.cc:482] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.392256 13842 raft_consensus.cc:504] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.393081 13842 leader_election.cc:288] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.393263 13842 raft_consensus.cc:482] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.393445 13842 raft_consensus.cc:504] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:44.393918 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_052f7d1eab54464d8743bd859c9f5bad" } modify_external_catalogs: true
I20240506 19:56:44.394134 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_052f7d1eab54464d8743bd859c9f5bad" } modify_external_catalogs: true
I20240506 19:56:44.394193 13842 leader_election.cc:288] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:44.394227 12781 leader_election.cc:341] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 4a080ccac6394210b0b24b65843f9d75
W20240506 19:56:44.395586 12781 leader_election.cc:341] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b38453bf678748399339806d12993c88
I20240506 19:56:44.413391 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 825528636a8f43188a863d2f28588a77 (DEFAULT_TABLE table=default.Table_20af0dc96de14fdf9a880466506fde22 [id=f9fd9f4f4e0f40ce8b953a79860fb234]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.413758 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 825528636a8f43188a863d2f28588a77 (DEFAULT_TABLE table=default.Table_20af0dc96de14fdf9a880466506fde22 [id=f9fd9f4f4e0f40ce8b953a79860fb234]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.414089 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825528636a8f43188a863d2f28588a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.415005 13249 tablet_bootstrap.cc:492] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.415174 13249 ts_tablet_manager.cc:1367] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.000s
I20240506 19:56:44.416188 13249 raft_consensus.cc:348] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.416622 13249 raft_consensus.cc:374] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.416725 13249 raft_consensus.cc:729] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.416942 13249 consensus_queue.cc:260] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.417167 13246 ts_tablet_manager.cc:1875] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.419756 13249 ts_tablet_manager.cc:1398] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:56:44.425328 13242 log.cc:1196] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/c5e3f798d6484d4fabd55f9b4c98a911
I20240506 19:56:44.426095 13848 raft_consensus.cc:482] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.426383 13848 raft_consensus.cc:504] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.427294 13848 leader_election.cc:288] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.429698 13246 ts_tablet_manager.cc:1888] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:44.429862 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 582725a8a48546a68937f7d11a1873a4 (DEFAULT_TABLE table=default.Table_1513820768504912ad5f7b9966ada614 [id=02a9361327124ac9a5359df5020f59e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.429973 13246 log.cc:1196] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2519c03738864f6ab5dcfab7bc3fc8aa
I20240506 19:56:44.430518 13246 ts_tablet_manager.cc:1909] T 2519c03738864f6ab5dcfab7bc3fc8aa P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:44.430948 12783 leader_election.cc:341] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b38453bf678748399339806d12993c88
I20240506 19:56:44.432222 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 582725a8a48546a68937f7d11a1873a4 (DEFAULT_TABLE table=default.Table_1513820768504912ad5f7b9966ada614 [id=02a9361327124ac9a5359df5020f59e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.434599 12783 leader_election.cc:302] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:44.436719 12783 leader_election.cc:341] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 4a080ccac6394210b0b24b65843f9d75
I20240506 19:56:44.437551 13243 ts_tablet_manager.cc:1888] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.437999 13243 log.cc:1196] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f0a8b12f36e74c95ad6eaf30e432f3e9
I20240506 19:56:44.438436 13872 raft_consensus.cc:2726] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:44.440438 13243 ts_tablet_manager.cc:1909] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.442569 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582725a8a48546a68937f7d11a1873a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.444962 12783 leader_election.cc:302] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:44.445399 13872 raft_consensus.cc:2726] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:44.449579 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1e22e5a015f74526b49363bc789cb4ed
I20240506 19:56:44.450564 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b5a443c0d35741b9966b32a7808a97e2
I20240506 19:56:44.451035 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 5b15189e5cb54684a0f62308c2e5f131 (DEFAULT_TABLE table=default.Table_491916e47bc640f199cb29f14c922469 [id=214553f8bf23497e841a4e1e364633f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.451346 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e22e5a015f74526b49363bc789cb4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.451596 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e22e5a015f74526b49363bc789cb4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.451951 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a443c0d35741b9966b32a7808a97e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.452260 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a443c0d35741b9966b32a7808a97e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.452611 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 582725a8a48546a68937f7d11a1873a4 (DEFAULT_TABLE table=default.Table_1513820768504912ad5f7b9966ada614 [id=02a9361327124ac9a5359df5020f59e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.452711 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:56:44.453153 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582725a8a48546a68937f7d11a1873a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.453572 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 30ad527c3f2a4b53affa928e149db1df
I20240506 19:56:44.453887 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.454470 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad527c3f2a4b53affa928e149db1df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.455024 12807 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 106) took 1512 ms. Trace:
I20240506 19:56:44.455466 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.456094 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad527c3f2a4b53affa928e149db1df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.456423 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 526) took 1177 ms. Trace:
I20240506 19:56:44.457650 13242 ts_tablet_manager.cc:1909] T c5e3f798d6484d4fabd55f9b4c98a911 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:44.457785 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b15189e5cb54684a0f62308c2e5f131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.458045 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 389) took 3597 ms. Trace:
I20240506 19:56:44.461869 12516 catalog_manager.cc:5484] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:44.462380 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:44.466051 13249 tablet_bootstrap.cc:492] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.467690 13249 tablet_bootstrap.cc:654] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.468883 12807 rpcz_store.cc:276] 0506 19:56:42.923036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.923131 (+ 95us) service_pool.cc:225] Handling call
0506 19:56:43.313054 (+389923us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x6160002d0830. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4e891628 00007f7b5128f768 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:44.435257 (+1122203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48112000}
I20240506 19:56:44.469126 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825528636a8f43188a863d2f28588a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.470243 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 491909887be54436bb94162ace8dd78e (DEFAULT_TABLE table=default.Table_b1f83bed002d4d3fbf1a4b5b95f0ea54 [id=6ae71958a02d48b384248581d7d91404]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.470669 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491909887be54436bb94162ace8dd78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.471056 13242 rpcz_store.cc:276] 0506 19:56:40.860793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.861160 (+ 367us) service_pool.cc:225] Handling call
0506 19:56:43.885701 (+3024541us) ts_tablet_manager.cc:1120] Deleting tablet c5e3f798d6484d4fabd55f9b4c98a911
0506 19:56:43.885712 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.016976 (+131264us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.377465 (+360489us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.458012 (+ 80547us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3024290}
I20240506 19:56:44.471486 13242 tablet_replica.cc:330] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:44.472041 13242 raft_consensus.cc:2229] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.472239 13242 raft_consensus.cc:2258] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.472395 12964 tablet_service.cc:1467] Processing CreateTablet for tablet f476f745e0784f7f9886484367dd8c22 (DEFAULT_TABLE table=default.Table_ef6c459b3eed45d08d709ab6f797aa4a [id=4c8cefa0eccb409fb8e582d47244e81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.472976 13242 ts_tablet_manager.cc:1875] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.473179 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476f745e0784f7f9886484367dd8c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.473241 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 2db006c642ce4db78c5b7355300e9567 (DEFAULT_TABLE table=default.Table_53cda80d706841ef876c3a0a4b51271a [id=a882030e617646b886caac521fb79161]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.473624 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db006c642ce4db78c5b7355300e9567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.478046 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 2919538141834fb6a83259e29288acdd (DEFAULT_TABLE table=default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.480752 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582725a8a48546a68937f7d11a1873a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.481135 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2919538141834fb6a83259e29288acdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.478891 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 825528636a8f43188a863d2f28588a77 (DEFAULT_TABLE table=default.Table_20af0dc96de14fdf9a880466506fde22 [id=f9fd9f4f4e0f40ce8b953a79860fb234]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.483970 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_109dfbf34cd845b3b626bc0a00a23e52" } new_table_name: "default.Table_5df015fc4a1f443e816d4e6b097fa6c0" modify_external_catalogs: true
I20240506 19:56:44.484274 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1698ae51beba4bd0a7c376126987543b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.485016 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1698ae51beba4bd0a7c376126987543b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.485118 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dfe68f19a2874be5bdd3c0321ba9872d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.485623 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfe68f19a2874be5bdd3c0321ba9872d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.485553 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cda4f793e543441f94c07714c44ad670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.486207 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cda4f793e543441f94c07714c44ad670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.486562 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a2fb551b2fc541e38f0a655c4539060b" } new_table_name: "default.Table_e091d2a1ea93426dacdda3d117da60b0" modify_external_catalogs: true
I20240506 19:56:44.487999 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.488052 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 5b15189e5cb54684a0f62308c2e5f131 (DEFAULT_TABLE table=default.Table_491916e47bc640f199cb29f14c922469 [id=214553f8bf23497e841a4e1e364633f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.488029 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a443c0d35741b9966b32a7808a97e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.489207 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b15189e5cb54684a0f62308c2e5f131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.489324 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e22e5a015f74526b49363bc789cb4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.489324 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5f73ddfe064b4a9ea98eaa48ce44334f" } new_table_name: "default.Table_94aeb54aa38948f7b903f8074dba8b52" modify_external_catalogs: true
I20240506 19:56:44.491334 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 2919538141834fb6a83259e29288acdd (DEFAULT_TABLE table=default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.491719 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2919538141834fb6a83259e29288acdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.490972 13246 rpcz_store.cc:276] 0506 19:56:43.269400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.270389 (+ 989us) service_pool.cc:225] Handling call
0506 19:56:44.268470 (+998081us) ts_tablet_manager.cc:1120] Deleting tablet 2519c03738864f6ab5dcfab7bc3fc8aa
0506 19:56:44.268479 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.420499 (+152020us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.428027 (+ 7528us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.447317 (+ 19290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997540}
I20240506 19:56:44.493925 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2519c03738864f6ab5dcfab7bc3fc8aa (table default.Table_23d6ef6eb15d489e813fdd5562e5be63 [id=5d61a15249c74e509342122448b76500]) successfully deleted
I20240506 19:56:44.494603 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.497386 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad527c3f2a4b53affa928e149db1df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.498092 13104 tablet_service.cc:1467] Processing CreateTablet for tablet f476f745e0784f7f9886484367dd8c22 (DEFAULT_TABLE table=default.Table_ef6c459b3eed45d08d709ab6f797aa4a [id=4c8cefa0eccb409fb8e582d47244e81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.498658 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476f745e0784f7f9886484367dd8c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.501668 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet c5e3f798d6484d4fabd55f9b4c98a911 (table default.Table_d08a9f7decde4bb69b28102c4b21a5bd [id=249eebb1852745929487a8fff7406ca1]) successfully deleted
I20240506 19:56:44.515144 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 7de7397db1e74b4488a3f9a70136bccb from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.522445 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f0a8b12f36e74c95ad6eaf30e432f3e9 (table default.Table_038664bd22da4e20b40b21843516298f [id=34aeae2e9b5546d1b405ce1432e3c0cb]) successfully deleted
I20240506 19:56:44.523473 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a8f2511323134e30aeb69565f63f961f" } new_table_name: "default.Table_b41fdc2ddabb43b6abd4d24fecd7447a" modify_external_catalogs: true
I20240506 19:56:44.523741 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3cd77b3dbc674e2099537f5b46662b86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.524571 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cd77b3dbc674e2099537f5b46662b86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.525604 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 2db006c642ce4db78c5b7355300e9567 (DEFAULT_TABLE table=default.Table_53cda80d706841ef876c3a0a4b51271a [id=a882030e617646b886caac521fb79161]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.526171 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db006c642ce4db78c5b7355300e9567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.527618 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 491909887be54436bb94162ace8dd78e (DEFAULT_TABLE table=default.Table_b1f83bed002d4d3fbf1a4b5b95f0ea54 [id=6ae71958a02d48b384248581d7d91404]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.531144 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491909887be54436bb94162ace8dd78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.535058 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 2db006c642ce4db78c5b7355300e9567 (DEFAULT_TABLE table=default.Table_53cda80d706841ef876c3a0a4b51271a [id=a882030e617646b886caac521fb79161]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.535171 13246 tablet_replica.cc:330] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:44.535459 13246 raft_consensus.cc:2229] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:44.536728 13246 pending_rounds.cc:62] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:44.536834 13246 pending_rounds.cc:68] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744056023580672 op_type: NO_OP noop_request { }
I20240506 19:56:44.537962 13246 raft_consensus.cc:2866] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:44.538154 13246 raft_consensus.cc:2258] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.538442 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 5b15189e5cb54684a0f62308c2e5f131 (DEFAULT_TABLE table=default.Table_491916e47bc640f199cb29f14c922469 [id=214553f8bf23497e841a4e1e364633f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.539191 13246 ts_tablet_manager.cc:1875] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.543802 13243 tablet_replica.cc:330] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.545442 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 2919538141834fb6a83259e29288acdd (DEFAULT_TABLE table=default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.546638 13249 tablet_bootstrap.cc:492] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.546808 13249 ts_tablet_manager.cc:1367] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.009s
I20240506 19:56:44.547646 13249 raft_consensus.cc:348] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.547888 13249 raft_consensus.cc:374] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.547801 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 491909887be54436bb94162ace8dd78e (DEFAULT_TABLE table=default.Table_b1f83bed002d4d3fbf1a4b5b95f0ea54 [id=6ae71958a02d48b384248581d7d91404]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.547953 13249 raft_consensus.cc:729] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.548179 12819 tablet_service.cc:1467] Processing CreateTablet for tablet f476f745e0784f7f9886484367dd8c22 (DEFAULT_TABLE table=default.Table_ef6c459b3eed45d08d709ab6f797aa4a [id=4c8cefa0eccb409fb8e582d47244e81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.548208 13249 consensus_queue.cc:260] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.548601 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a8b12f36e74c95ad6eaf30e432f3e9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.551498 13058 leader_election.cc:341] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.551538 13249 ts_tablet_manager.cc:1398] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:44.552278 13249 tablet_bootstrap.cc:492] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.555976 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db006c642ce4db78c5b7355300e9567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.556196 13249 tablet_bootstrap.cc:654] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.557852 13872 raft_consensus.cc:482] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.559094 13872 raft_consensus.cc:504] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.560660 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491909887be54436bb94162ace8dd78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.561885 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825528636a8f43188a863d2f28588a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.565061 13243 raft_consensus.cc:2229] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.568344 13124 raft_consensus.cc:1261] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:44.568998 13864 consensus_queue.cc:1035] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:44.572118 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476f745e0784f7f9886484367dd8c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.576303 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c895888dd3b4499a990be1065b5cdbc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.576895 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c895888dd3b4499a990be1065b5cdbc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.583721 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.585755 13243 raft_consensus.cc:2258] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.586063 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b15189e5cb54684a0f62308c2e5f131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.586901 13872 leader_election.cc:288] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.587072 13243 ts_tablet_manager.cc:1875] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.588320 13249 tablet_bootstrap.cc:492] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
W20240506 19:56:44.589576 12781 leader_election.cc:341] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:44.590054 13249 ts_tablet_manager.cc:1367] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
W20240506 19:56:44.590270 12921 consensus_peers.cc:485] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:44.591061 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2919538141834fb6a83259e29288acdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.590971 13249 raft_consensus.cc:348] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.591208 13249 raft_consensus.cc:374] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.591293 13249 raft_consensus.cc:729] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.591574 13249 consensus_queue.cc:260] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.592231 13249 ts_tablet_manager.cc:1398] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:44.592514 13249 tablet_bootstrap.cc:492] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:44.593677 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c96f014293242be9640515697c0733b consensus state has no leader
W20240506 19:56:44.595332 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8321457717fd4ad28ae1292d9e9a8669 consensus state has no leader
W20240506 19:56:44.600189 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 312673us
W20240506 19:56:44.600383 12783 leader_election.cc:341] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:44.600654 12783 leader_election.cc:302] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:44.601946 13878 raft_consensus.cc:482] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.601994 13872 raft_consensus.cc:2726] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:44.603749 13057 leader_election.cc:341] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f0a8b12f36e74c95ad6eaf30e432f3e9
I20240506 19:56:44.606561 13057 leader_election.cc:302] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.608191 13243 ts_tablet_manager.cc:1888] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.608326 13243 log.cc:1196] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d05c5ca7ff864fa98f3494032729e978
I20240506 19:56:44.608527 13877 raft_consensus.cc:2726] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:44.610050 13249 tablet_bootstrap.cc:654] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.611143 13243 ts_tablet_manager.cc:1909] T d05c5ca7ff864fa98f3494032729e978 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.612203 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d05c5ca7ff864fa98f3494032729e978 (table default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]) successfully deleted
I20240506 19:56:44.613669 13243 tablet_replica.cc:330] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.614070 13243 raft_consensus.cc:2229] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:44.614729 13243 raft_consensus.cc:2258] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.615074 13242 ts_tablet_manager.cc:1888] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:44.615293 13242 log.cc:1196] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/5dc0ac37f67942fc8a31d8d43c839246
I20240506 19:56:44.615792 13242 ts_tablet_manager.cc:1909] T 5dc0ac37f67942fc8a31d8d43c839246 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:44.615810 13243 ts_tablet_manager.cc:1875] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.610489 13878 raft_consensus.cc:504] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.629958 13243 ts_tablet_manager.cc:1888] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:44.631275 13243 log.cc:1196] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/acd3fc62e9614cf3bde0202ee6832bb3
I20240506 19:56:44.631734 13243 ts_tablet_manager.cc:1909] T acd3fc62e9614cf3bde0202ee6832bb3 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.632911 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]) successfully deleted
I20240506 19:56:44.634507 13245 tablet_bootstrap.cc:492] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:44.635541 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1e22e5a015f74526b49363bc789cb4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e22e5a015f74526b49363bc789cb4ed already in progress: creating tablet
I20240506 19:56:44.635715 13245 ts_tablet_manager.cc:1367] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.360s user 0.050s sys 0.057s
I20240506 19:56:44.635699 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e7329b22414da886dee4e1ec6bf3ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:44.635928 12988 raft_consensus.cc:2454] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:44.636135 13243 tablet_replica.cc:330] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.636358 13243 raft_consensus.cc:2229] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.636381 13057 leader_election.cc:302] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.636723 13877 raft_consensus.cc:2781] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:44.636662 13245 raft_consensus.cc:348] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.636816 13877 raft_consensus.cc:482] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:44.636888 13245 raft_consensus.cc:374] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.636906 13877 raft_consensus.cc:3037] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.636917 13243 raft_consensus.cc:2258] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.636988 13245 raft_consensus.cc:729] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.637235 13878 leader_election.cc:288] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.637718 13243 ts_tablet_manager.cc:1875] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.638329 13245 consensus_queue.cc:260] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.638883 13245 ts_tablet_manager.cc:1398] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:44.639896 13246 ts_tablet_manager.cc:1888] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:44.640040 13246 log.cc:1196] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d05c5ca7ff864fa98f3494032729e978
I20240506 19:56:44.639984 13877 raft_consensus.cc:504] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.640488 13246 ts_tablet_manager.cc:1909] T d05c5ca7ff864fa98f3494032729e978 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:44.642289 13877 leader_election.cc:288] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.643975 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e7329b22414da886dee4e1ec6bf3ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:44.644191 12988 raft_consensus.cc:3037] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.646370 13245 tablet_bootstrap.cc:492] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:44.646975 12988 raft_consensus.cc:2454] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:44.648131 13057 leader_election.cc:302] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:44.649565 13246 tablet_replica.cc:330] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:44.649885 13246 raft_consensus.cc:2229] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.650152 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d05c5ca7ff864fa98f3494032729e978 (table default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]) successfully deleted
I20240506 19:56:44.651203 13246 raft_consensus.cc:2258] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.652557 13246 ts_tablet_manager.cc:1875] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.654742 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_114d9e27c6d74425802c46e8610d3343"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.656610 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_114d9e27c6d74425802c46e8610d3343 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.630738 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e7329b22414da886dee4e1ec6bf3ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:44.642578 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e7329b22414da886dee4e1ec6bf3ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:44.658308 13058 leader_election.cc:341] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:44.658656 13058 leader_election.cc:341] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.658496 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_330c25efd3544e359efe8b4e731ce71c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.659111 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_330c25efd3544e359efe8b4e731ce71c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.659322 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 390) took 3782 ms. Trace:
I20240506 19:56:44.659428 13242 rpcz_store.cc:276] 0506 19:56:40.877251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.877331 (+ 80us) service_pool.cc:225] Handling call
0506 19:56:44.471408 (+3594077us) ts_tablet_manager.cc:1120] Deleting tablet 5dc0ac37f67942fc8a31d8d43c839246
0506 19:56:44.471419 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.487067 (+ 15648us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.614914 (+127847us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.659270 (+ 44356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3587297}
I20240506 19:56:44.659914 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 392) took 3741 ms. Trace:
I20240506 19:56:44.667078 13242 rpcz_store.cc:276] 0506 19:56:40.917977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.918020 (+ 43us) service_pool.cc:225] Handling call
0506 19:56:44.659770 (+3741750us) ts_tablet_manager.cc:1120] Deleting tablet 034d6309de0740d0a5806b3847e334cb
0506 19:56:44.659780 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.659869 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3741499}
I20240506 19:56:44.670212 13243 ts_tablet_manager.cc:1888] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.670409 13243 log.cc:1196] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b5a443c0d35741b9966b32a7808a97e2
I20240506 19:56:44.670836 13243 ts_tablet_manager.cc:1909] T b5a443c0d35741b9966b32a7808a97e2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.672048 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b5a443c0d35741b9966b32a7808a97e2 (table default.Table_1a5485067c1a4e529059f5434babbdc7 [id=54286945a7554349b48aa6511f84326c]) successfully deleted
W20240506 19:56:44.672345 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aa8f9aaca444c839a88c07f58cb6ad6 already in progress: creating tablet
I20240506 19:56:44.674235 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]). Expected version 3 got 0
I20240506 19:56:44.674454 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 391) took 3756 ms. Trace:
I20240506 19:56:44.675009 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0c42cb0cd7e349fa86150517492387bb" } modify_external_catalogs: true
I20240506 19:56:44.675284 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0c42cb0cd7e349fa86150517492387bb" } modify_external_catalogs: true
I20240506 19:56:44.679769 13243 tablet_replica.cc:330] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.680807 13246 ts_tablet_manager.cc:1888] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.681329 13246 log.cc:1196] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f3e7329b22414da886dee4e1ec6bf3ad
I20240506 19:56:44.681535 13243 raft_consensus.cc:2229] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.681425 13242 rpcz_store.cc:276] 0506 19:56:40.917854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.920466 (+ 2612us) service_pool.cc:225] Handling call
0506 19:56:44.674264 (+3753798us) ts_tablet_manager.cc:1120] Deleting tablet f8ffb88e0a0c4125b04f4d8d328da3f5
0506 19:56:44.674276 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.674387 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3753473}
W20240506 19:56:44.680224 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 30ad527c3f2a4b53affa928e149db1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ad527c3f2a4b53affa928e149db1df already in progress: creating tablet
I20240506 19:56:44.682124 13243 raft_consensus.cc:2258] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.682709 13246 ts_tablet_manager.cc:1909] T f3e7329b22414da886dee4e1ec6bf3ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:44.682860 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8ffb88e0a0c4125b04f4d8d328da3f5 already in progress: creating tablet
I20240506 19:56:44.683450 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 393) took 3706 ms. Trace:
I20240506 19:56:44.683667 13243 ts_tablet_manager.cc:1875] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.685256 13242 rpcz_store.cc:276] 0506 19:56:40.976867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.978521 (+ 1654us) service_pool.cc:225] Handling call
0506 19:56:44.683279 (+3704758us) ts_tablet_manager.cc:1120] Deleting tablet bece304e0ba048ecbb2d92aa71fdf707
0506 19:56:44.683288 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.683395 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3704526}
I20240506 19:56:44.685539 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f3e7329b22414da886dee4e1ec6bf3ad (table default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc [id=49660c755f8f470b8b8dc301ca27bb26]) successfully deleted
W20240506 19:56:44.688231 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bece304e0ba048ecbb2d92aa71fdf707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bece304e0ba048ecbb2d92aa71fdf707 already in progress: creating tablet
I20240506 19:56:44.688868 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 394) took 3711 ms. Trace:
W20240506 19:56:44.689625 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea7e073296ce46e786362aaed96eb4dd consensus state has no leader
I20240506 19:56:44.689818 13242 rpcz_store.cc:276] 0506 19:56:40.977002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.978782 (+ 1780us) service_pool.cc:225] Handling call
0506 19:56:44.688670 (+3709888us) ts_tablet_manager.cc:1120] Deleting tablet c1ff7a3638004012b08c973192c055a5
0506 19:56:44.688684 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.688813 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3709738}
W20240506 19:56:44.690207 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8a45a853e310473d844a423fbefc553a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a45a853e310473d844a423fbefc553a already in progress: creating tablet
I20240506 19:56:44.690209 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fb159d58ea054e2883302a7bc6ea228b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.690935 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c1ff7a3638004012b08c973192c055a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1ff7a3638004012b08c973192c055a5 already in progress: creating tablet
W20240506 19:56:44.690990 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb159d58ea054e2883302a7bc6ea228b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:44.691251 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 65da8b15e0c6496bb86b47c49e348af0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65da8b15e0c6496bb86b47c49e348af0 already in progress: creating tablet
I20240506 19:56:44.691653 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 395) took 3714 ms. Trace:
I20240506 19:56:44.693356 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_58aa959f517f46b6a0c3a485458211bd" } new_table_name: "default.Table_294ef89cfe854b01a260a32748b84c85" modify_external_catalogs: true
W20240506 19:56:44.695647 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 452a83470585433e8def7174300d3a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452a83470585433e8def7174300d3a7f already in progress: creating tablet
W20240506 19:56:44.695953 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2ff3435c11854649b7daf74f3d8c33d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ff3435c11854649b7daf74f3d8c33d7 already in progress: creating tablet
W20240506 19:56:44.696166 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 08dafb661b2443f5afadc6816868bac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08dafb661b2443f5afadc6816868bac8 already in progress: creating tablet
I20240506 19:56:44.696372 13246 tablet_replica.cc:330] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:44.696408 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 880ecc75bab446e39c687f3c8b740361 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 880ecc75bab446e39c687f3c8b740361 already in progress: creating tablet
I20240506 19:56:44.697074 13246 raft_consensus.cc:2229] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.697331 13246 raft_consensus.cc:2258] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.697929 13246 ts_tablet_manager.cc:1875] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.698184 13242 rpcz_store.cc:276] 0506 19:56:40.977081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.978951 (+ 1870us) service_pool.cc:225] Handling call
0506 19:56:44.691464 (+3712513us) ts_tablet_manager.cc:1120] Deleting tablet 373e04e93a5543f9934077006eaf2ba9
0506 19:56:44.691477 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.691600 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3712337}
W20240506 19:56:44.701387 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 373e04e93a5543f9934077006eaf2ba9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 373e04e93a5543f9934077006eaf2ba9 already in progress: creating tablet
I20240506 19:56:44.702011 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 396) took 3724 ms. Trace:
I20240506 19:56:44.704322 13245 tablet_bootstrap.cc:654] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.712493 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8b6089202064458fa076b29994bd285e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.713040 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b6089202064458fa076b29994bd285e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.716177 13242 rpcz_store.cc:276] 0506 19:56:40.977238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:40.979147 (+ 1909us) service_pool.cc:225] Handling call
0506 19:56:44.701792 (+3722645us) ts_tablet_manager.cc:1120] Deleting tablet 1a6e0126d9854215a0e174a76c0a0391
0506 19:56:44.701805 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.701944 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722494}
I20240506 19:56:44.719763 13242 tablet_replica.cc:330] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:44.720309 13242 raft_consensus.cc:2229] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.722389 13243 ts_tablet_manager.cc:1888] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.722568 13243 log.cc:1196] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:44.724231 13242 raft_consensus.cc:2258] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.724308 13243 ts_tablet_manager.cc:1909] T 20c1590e36b4471d8e72f4b07410ee0f P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:44.725191 13242 ts_tablet_manager.cc:1875] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.729229 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e0368ab9f3a24d2ea6b743689cb3a9b7" } modify_external_catalogs: true
I20240506 19:56:44.729537 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e0368ab9f3a24d2ea6b743689cb3a9b7" } modify_external_catalogs: true
I20240506 19:56:44.723886 13244 tablet_bootstrap.cc:492] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:44.729858 13244 ts_tablet_manager.cc:1367] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.572s user 0.017s sys 0.023s
W20240506 19:56:44.730022 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99f4861a01404663870ce91978c8af4f consensus state has no leader
I20240506 19:56:44.730718 13244 raft_consensus.cc:348] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.732662 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_052649a7995c41f2a0dde09686115a96" } modify_external_catalogs: true
I20240506 19:56:44.732867 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_052649a7995c41f2a0dde09686115a96" } modify_external_catalogs: true
I20240506 19:56:44.733386 13244 raft_consensus.cc:374] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.733502 13244 raft_consensus.cc:729] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.733841 13244 consensus_queue.cc:260] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.742000 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 20c1590e36b4471d8e72f4b07410ee0f (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:44.743441 13903 tablet.cc:1722] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:44.748629 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 2751074a54ed4bcea6979e6ad11c2efe with delete_type TABLET_DATA_DELETED (replaced by tablet 45b2a4720f6a46e4b0412a58c1e26bf7) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.748905 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7de7397db1e74b4488a3f9a70136bccb
I20240506 19:56:44.749027 13902 tablet.cc:1722] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:44.749490 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 7de7397db1e74b4488a3f9a70136bccb deleted and replaced by tablet 65315d8ac5ff408bae8593681335e5b1
I20240506 19:56:44.752352 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.752564 13244 ts_tablet_manager.cc:1398] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
W20240506 19:56:44.753067 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85365758f6a64536ab61e8349de63dea consensus state has no leader
I20240506 19:56:44.754571 13244 tablet_bootstrap.cc:492] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:44.755949 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7397db1e74b4488a3f9a70136bccb with delete_type TABLET_DATA_DELETED (replaced by tablet 65315d8ac5ff408bae8593681335e5b1) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.756278 13244 tablet_bootstrap.cc:654] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.757534 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7397db1e74b4488a3f9a70136bccb with delete_type TABLET_DATA_DELETED (replaced by tablet 65315d8ac5ff408bae8593681335e5b1) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.762585 13245 tablet_bootstrap.cc:492] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:44.762939 13245 ts_tablet_manager.cc:1367] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.000s
I20240506 19:56:44.764390 13245 raft_consensus.cc:348] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.764910 13245 raft_consensus.cc:374] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.765087 13245 raft_consensus.cc:729] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.765734 13245 consensus_queue.cc:260] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.767439 13245 ts_tablet_manager.cc:1398] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:44.770108 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7397db1e74b4488a3f9a70136bccb with delete_type TABLET_DATA_DELETED (replaced by tablet 65315d8ac5ff408bae8593681335e5b1) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.774267 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:44.775353 13353 raft_consensus.cc:482] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.775539 13353 raft_consensus.cc:504] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.776316 13353 leader_election.cc:288] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.776988 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:44.777240 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:44.777931 12921 leader_election.cc:341] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.774679 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_574f459d109b430ea8c288af9280052f" } modify_external_catalogs: true
I20240506 19:56:44.780982 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_574f459d109b430ea8c288af9280052f" } modify_external_catalogs: true
I20240506 19:56:44.776314 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_76271c0b874e41b29c550ad9d849d913" } modify_external_catalogs: true
I20240506 19:56:44.781561 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_76271c0b874e41b29c550ad9d849d913" } modify_external_catalogs: true
I20240506 19:56:44.782366 13249 tablet_bootstrap.cc:492] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.782567 13249 ts_tablet_manager.cc:1367] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.190s user 0.015s sys 0.036s
I20240506 19:56:44.783396 13249 raft_consensus.cc:348] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.783618 13249 raft_consensus.cc:374] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.783692 13249 raft_consensus.cc:729] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.783838 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_54e2ae0fb2324776911994a064b540f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:44.783943 13249 consensus_queue.cc:260] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:44.784487 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54e2ae0fb2324776911994a064b540f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.785493 13249 ts_tablet_manager.cc:1398] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:44.788468 12521 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet cd2a9bd7f83c44f295b6cbffc03245ed
I20240506 19:56:44.798377 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f249dc790e774c46a2cf5e2af4f2a7a5" } new_table_name: "default.Table_f68578ef105f4a5b8170783670e18751" modify_external_catalogs: true
I20240506 19:56:44.798432 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_799d22eaba644b6ea3acc774f66b8ba2" } new_table_name: "default.Table_2ba27e19baf9482da781cb704b29bb2b" modify_external_catalogs: true
I20240506 19:56:44.803517 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet b529bb9362924caca399491e7d9e03cd from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.813880 13902 tablet.cc:1722] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:44.814595 13122 raft_consensus.cc:2454] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:44.816479 12922 leader_election.cc:302] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.817008 13864 raft_consensus.cc:2781] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:44.817112 13864 raft_consensus.cc:482] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:44.817224 13864 raft_consensus.cc:3037] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.814417 13243 tablet_replica.cc:330] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:44.819981 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 9df3ab9dff5148f9bb335a7209b29bbc from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.820150 13249 tablet_bootstrap.cc:492] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.821920 13249 tablet_bootstrap.cc:654] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.825873 13243 raft_consensus.cc:2229] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.826459 13246 ts_tablet_manager.cc:1888] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.827039 13243 raft_consensus.cc:2258] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.828392 13244 tablet_bootstrap.cc:492] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:44.828639 13244 ts_tablet_manager.cc:1367] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.003s
I20240506 19:56:44.829455 13244 raft_consensus.cc:348] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.829778 13244 raft_consensus.cc:374] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.829923 13244 raft_consensus.cc:729] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.830250 13244 consensus_queue.cc:260] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.834964 13244 ts_tablet_manager.cc:1398] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:44.836634 13244 tablet_bootstrap.cc:492] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:44.836783 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6089039b55f466f84f27596d4c626ae consensus state has no leader
W20240506 19:56:44.837262 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 902d43d15d334939bddb254ae04ca174 consensus state has no leader
I20240506 19:56:44.837386 13246 log.cc:1196] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:44.837867 13246 ts_tablet_manager.cc:1909] T 07755e37a20646a3b1a7b900d54a077c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:44.824003 13245 tablet_bootstrap.cc:492] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:44.832731 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.832767 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.839627 13244 tablet_bootstrap.cc:654] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.840107 13245 tablet_bootstrap.cc:654] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.841066 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 (DEFAULT_TABLE table=default.Table_d7e3f13ea1c1428b9ca232be80f34d5b [id=cb31aaff53494e62b45e59ec237acd16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.841599 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 (DEFAULT_TABLE table=default.Table_d7e3f13ea1c1428b9ca232be80f34d5b [id=cb31aaff53494e62b45e59ec237acd16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.828732 13249 tablet_bootstrap.cc:492] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.839254 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 (DEFAULT_TABLE table=default.Table_d7e3f13ea1c1428b9ca232be80f34d5b [id=cb31aaff53494e62b45e59ec237acd16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.842139 13249 ts_tablet_manager.cc:1367] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240506 19:56:44.843037 13249 raft_consensus.cc:348] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.843241 13249 raft_consensus.cc:374] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.843348 13249 raft_consensus.cc:729] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.843647 13249 consensus_queue.cc:260] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.844014 12820 tablet_service.cc:1467] Processing CreateTablet for tablet b6089039b55f466f84f27596d4c626ae (DEFAULT_TABLE table=default.Table_2ba27e19baf9482da781cb704b29bb2b [id=9f92215443d6470f8ff3e7d27cbbf7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.845238 13093 tablet_service.cc:1467] Processing CreateTablet for tablet a564c22b68f74b92ba0f313fcb29c2f7 (DEFAULT_TABLE table=default.Table_70f25b0c770549f5b47f5ac2d70cd33c [id=ea59313ca58f41779f4e4fcc49144082]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.845889 13104 tablet_service.cc:1467] Processing CreateTablet for tablet b6089039b55f466f84f27596d4c626ae (DEFAULT_TABLE table=default.Table_2ba27e19baf9482da781cb704b29bb2b [id=9f92215443d6470f8ff3e7d27cbbf7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.846160 12819 tablet_service.cc:1467] Processing CreateTablet for tablet a564c22b68f74b92ba0f313fcb29c2f7 (DEFAULT_TABLE table=default.Table_70f25b0c770549f5b47f5ac2d70cd33c [id=ea59313ca58f41779f4e4fcc49144082]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.846635 12817 tablet_service.cc:1467] Processing CreateTablet for tablet e7f36b24ff9848a18e23a599a72ffae2 (DEFAULT_TABLE table=default.Table_19ad839886424d5baf4207f64dbfbd8b [id=e341934744774a2eb8baf489109c16f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.847234 13092 tablet_service.cc:1467] Processing CreateTablet for tablet e7f36b24ff9848a18e23a599a72ffae2 (DEFAULT_TABLE table=default.Table_19ad839886424d5baf4207f64dbfbd8b [id=e341934744774a2eb8baf489109c16f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.847719 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 054d48760d684adeb8196b0532b07101 (DEFAULT_TABLE table=default.Table_cc43c588b4804f9aa2bbadd0d65ce064 [id=80efe594c8b64a04864ac3a11061edd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.847785 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 054d48760d684adeb8196b0532b07101 (DEFAULT_TABLE table=default.Table_cc43c588b4804f9aa2bbadd0d65ce064 [id=80efe594c8b64a04864ac3a11061edd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.848287 12962 tablet_service.cc:1467] Processing CreateTablet for tablet b6089039b55f466f84f27596d4c626ae (DEFAULT_TABLE table=default.Table_2ba27e19baf9482da781cb704b29bb2b [id=9f92215443d6470f8ff3e7d27cbbf7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.848793 12958 tablet_service.cc:1467] Processing CreateTablet for tablet a564c22b68f74b92ba0f313fcb29c2f7 (DEFAULT_TABLE table=default.Table_70f25b0c770549f5b47f5ac2d70cd33c [id=ea59313ca58f41779f4e4fcc49144082]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.849035 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 7c40ba83b107436f803b11ae90f1b980 (DEFAULT_TABLE table=default.Table_7511af3d69ea4e9d921a500168622d24 [id=56e200eedf3e4fdfa122be9f5cc65bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.849510 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 054d48760d684adeb8196b0532b07101 (DEFAULT_TABLE table=default.Table_cc43c588b4804f9aa2bbadd0d65ce064 [id=80efe594c8b64a04864ac3a11061edd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.849974 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 7c40ba83b107436f803b11ae90f1b980 (DEFAULT_TABLE table=default.Table_7511af3d69ea4e9d921a500168622d24 [id=56e200eedf3e4fdfa122be9f5cc65bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.849218 12964 tablet_service.cc:1467] Processing CreateTablet for tablet e7f36b24ff9848a18e23a599a72ffae2 (DEFAULT_TABLE table=default.Table_19ad839886424d5baf4207f64dbfbd8b [id=e341934744774a2eb8baf489109c16f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.850579 13243 ts_tablet_manager.cc:1875] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.851336 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 7c40ba83b107436f803b11ae90f1b980 (DEFAULT_TABLE table=default.Table_7511af3d69ea4e9d921a500168622d24 [id=56e200eedf3e4fdfa122be9f5cc65bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:44.854933 13242 ts_tablet_manager.cc:1888] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:44.855101 13242 log.cc:1196] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/6d8a9ecf47d7474c82758417449d940c
I20240506 19:56:44.855269 13903 tablet.cc:1722] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:44.855489 13242 ts_tablet_manager.cc:1909] T 6d8a9ecf47d7474c82758417449d940c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:44.855861 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec1ac39f62b4e15a267f5db2a6166c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.856470 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c40ba83b107436f803b11ae90f1b980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.856412 13864 raft_consensus.cc:504] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.856961 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f36b24ff9848a18e23a599a72ffae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.857187 13864 leader_election.cc:288] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.857674 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:44.858376 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec1ac39f62b4e15a267f5db2a6166c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:44.859162 12921 leader_election.cc:341] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.859265 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054d48760d684adeb8196b0532b07101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.859527 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec1ac39f62b4e15a267f5db2a6166c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.859794 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a564c22b68f74b92ba0f313fcb29c2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.860046 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a564c22b68f74b92ba0f313fcb29c2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.860311 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c40ba83b107436f803b11ae90f1b980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.860466 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f36b24ff9848a18e23a599a72ffae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.860833 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054d48760d684adeb8196b0532b07101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.860978 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a564c22b68f74b92ba0f313fcb29c2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.861275 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c40ba83b107436f803b11ae90f1b980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.861569 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054d48760d684adeb8196b0532b07101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.861687 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6089039b55f466f84f27596d4c626ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.862573 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6089039b55f466f84f27596d4c626ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.863001 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6089039b55f466f84f27596d4c626ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.865490 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f36b24ff9848a18e23a599a72ffae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:44.871831 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 546) took 1043 ms. Trace:
I20240506 19:56:44.871968 13246 rpcz_store.cc:276] 0506 19:56:43.828454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.828508 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:44.696314 (+867806us) ts_tablet_manager.cc:1120] Deleting tablet 07755e37a20646a3b1a7b900d54a077c
0506 19:56:44.696320 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.718306 (+ 21986us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.826305 (+107999us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.871765 (+ 45460us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":867600}
I20240506 19:56:44.872247 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f] alter complete (version 1)
I20240506 19:56:44.873009 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 07755e37a20646a3b1a7b900d54a077c (table default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]) successfully deleted
I20240506 19:56:44.873097 13246 tablet_replica.cc:330] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:44.873315 13246 raft_consensus.cc:2229] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:44.873329 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 986f9c83b04f41b49e9b78fce731a90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 986f9c83b04f41b49e9b78fce731a90f already in progress: creating tablet
I20240506 19:56:44.873534 13246 raft_consensus.cc:2258] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:44.873566 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c652feb405ef4f1cab8ee602e9d3eb18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c652feb405ef4f1cab8ee602e9d3eb18 already in progress: creating tablet
W20240506 19:56:44.873813 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d19db9cff74843e087fd82db2475c29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d19db9cff74843e087fd82db2475c29c already in progress: creating tablet
I20240506 19:56:44.874104 13246 ts_tablet_manager.cc:1875] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:44.874459 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ee62b918bba543a6b3e467fd8dc59878 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee62b918bba543a6b3e467fd8dc59878 already in progress: creating tablet
W20240506 19:56:44.874749 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 40b9731a473449b19a9e95653a9d0830 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40b9731a473449b19a9e95653a9d0830 already in progress: creating tablet
I20240506 19:56:44.875375 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:44.875648 13124 raft_consensus.cc:3037] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.878311 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3dbe1bd9a46f4c9b8146d1be89e93c9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.878974 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dbe1bd9a46f4c9b8146d1be89e93c9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.881343 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:44.886999 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 1e22e5a015f74526b49363bc789cb4ed from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:44.889734 13249 ts_tablet_manager.cc:1398] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.047s user 0.003s sys 0.004s
I20240506 19:56:44.890190 13249 tablet_bootstrap.cc:492] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.892869 13249 tablet_bootstrap.cc:654] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.894120 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_438b764bbb844c7cbc052c23c33795ec" } new_table_name: "default.Table_3d6b94b6306c4200bfde515a4b2c418c" modify_external_catalogs: true
I20240506 19:56:44.897477 13245 tablet_bootstrap.cc:492] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:44.897693 13245 ts_tablet_manager.cc:1367] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.002s
I20240506 19:56:44.898818 13245 raft_consensus.cc:348] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.899066 13245 raft_consensus.cc:374] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.899181 13245 raft_consensus.cc:729] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.899279 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a8f8c4d749444684a5b327c80c5f5829"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:44.899459 13245 consensus_queue.cc:260] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:44.899924 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8f8c4d749444684a5b327c80c5f5829 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.900117 13245 ts_tablet_manager.cc:1398] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:44.900431 13245 tablet_bootstrap.cc:492] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:44.902129 13245 tablet_bootstrap.cc:654] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.905532 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 397) took 3876 ms. Trace:
I20240506 19:56:44.905776 13242 rpcz_store.cc:276] 0506 19:56:41.028952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.029018 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:44.719664 (+3690646us) ts_tablet_manager.cc:1120] Deleting tablet 6d8a9ecf47d7474c82758417449d940c
0506 19:56:44.719676 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.776075 (+ 56399us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.854754 (+ 78679us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:44.905481 (+ 50727us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3686941}
I20240506 19:56:44.905953 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 04738def56324a368f8182543107e32e
I20240506 19:56:44.906360 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 6d8a9ecf47d7474c82758417449d940c (table default.Table_8de44207890c4ceb8542f805a6f9b8da [id=70fc9e614eab4b11baeefbbf7127f540]) successfully deleted
I20240506 19:56:44.907016 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 04738def56324a368f8182543107e32e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.907107 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 04738def56324a368f8182543107e32e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.907449 13242 tablet_replica.cc:330] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:44.907598 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 03df4632adfa4b0ab999aeaa5bb4e1f4
I20240506 19:56:44.908277 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.907769 13242 raft_consensus.cc:2229] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:44.908624 13242 raft_consensus.cc:2258] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:44.908654 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.909535 13242 ts_tablet_manager.cc:1875] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:44.910251 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.910916 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:44.910938 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 04738def56324a368f8182543107e32e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.916806 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.917551 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.919526 13249 tablet_bootstrap.cc:492] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.919720 13249 ts_tablet_manager.cc:1367] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240506 19:56:44.920534 13249 raft_consensus.cc:348] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.920789 13249 raft_consensus.cc:374] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.920871 13249 raft_consensus.cc:729] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.921154 13249 consensus_queue.cc:260] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.921722 13249 ts_tablet_manager.cc:1398] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:44.922982 13249 tablet_bootstrap.cc:492] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:44.926370 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 50ef5a2799b447f992207452dbe39645
I20240506 19:56:44.927538 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ef5a2799b447f992207452dbe39645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.927827 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 09cb71b5962f4561bf0fdbe12bf45296
I20240506 19:56:44.928869 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 09cb71b5962f4561bf0fdbe12bf45296 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.928917 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 09cb71b5962f4561bf0fdbe12bf45296 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.929643 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ca23d6f0a9e0445cbd4d0e708c8de2d3
I20240506 19:56:44.930706 13249 tablet_bootstrap.cc:654] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.930940 12518 catalog_manager.cc:6554] ReplaceTablet: tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 deleted and replaced by tablet 9a8c77e550ef40779b9c64d89df40a9a
I20240506 19:56:44.931607 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a8c77e550ef40779b9c64d89df40a9a) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.932219 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a8c77e550ef40779b9c64d89df40a9a) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.933694 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 09cb71b5962f4561bf0fdbe12bf45296 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.933892 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a8c77e550ef40779b9c64d89df40a9a) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.933722 13245 tablet_bootstrap.cc:492] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:44.934213 13245 ts_tablet_manager.cc:1367] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240506 19:56:44.934180 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_680d6530958740e293cd6eddcad88b21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.934850 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_680d6530958740e293cd6eddcad88b21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.935151 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ef5a2799b447f992207452dbe39645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.935308 13245 raft_consensus.cc:348] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.935607 13245 raft_consensus.cc:374] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.935743 13245 raft_consensus.cc:729] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.936106 13245 consensus_queue.cc:260] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.936275 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9362c95fb2df44f88a0a579b79b4ddd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.936892 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9362c95fb2df44f88a0a579b79b4ddd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.937479 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ef5a2799b447f992207452dbe39645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.937813 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ca9a4ec248934294830f719bc5d46204" } new_table_name: "default.Table_71d092610e224787956867c83b11d4d7" modify_external_catalogs: true
I20240506 19:56:44.940541 13245 ts_tablet_manager.cc:1398] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:44.942963 13245 tablet_bootstrap.cc:492] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:44.947062 13911 raft_consensus.cc:482] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.947288 13911 raft_consensus.cc:504] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.947907 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:44.948434 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 9df3ab9dff5148f9bb335a7209b29bbc deleted and replaced by tablet efc41afd84a74652a82cc639bf562d99
I20240506 19:56:44.948804 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00de09ddcdd948a6a3500a6c9d814c14" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.949438 12921 leader_election.cc:341] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.949802 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00de09ddcdd948a6a3500a6c9d814c14" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:44.950086 13122 raft_consensus.cc:2454] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:44.950706 12922 leader_election.cc:302] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.951411 13864 raft_consensus.cc:2781] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:44.951534 13864 raft_consensus.cc:482] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:44.951628 13864 raft_consensus.cc:3037] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.952229 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc with delete_type TABLET_DATA_DELETED (replaced by tablet efc41afd84a74652a82cc639bf562d99) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:44.953883 13124 raft_consensus.cc:2454] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:44.954653 13864 raft_consensus.cc:504] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.956069 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00de09ddcdd948a6a3500a6c9d814c14" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:44.956673 12921 leader_election.cc:341] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.957480 13864 leader_election.cc:288] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.958809 12922 leader_election.cc:302] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.958874 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2fda57476820461185abfb4e54f9a3d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.959514 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fda57476820461185abfb4e54f9a3d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.959753 13864 raft_consensus.cc:2781] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:44.961064 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00de09ddcdd948a6a3500a6c9d814c14" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:44.961335 13124 raft_consensus.cc:3037] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:44.965603 13864 raft_consensus.cc:686] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:44.965791 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_07709e08656c44c892d8a4f181d10bd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:44.966471 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07709e08656c44c892d8a4f181d10bd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.966933 13864 consensus_queue.cc:237] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.968286 13911 leader_election.cc:288] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:44.968720 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc with delete_type TABLET_DATA_DELETED (replaced by tablet efc41afd84a74652a82cc639bf562d99) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.969065 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc with delete_type TABLET_DATA_DELETED (replaced by tablet efc41afd84a74652a82cc639bf562d99) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:44.976447 13878 raft_consensus.cc:482] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:44.976673 13878 raft_consensus.cc:504] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.977463 13878 leader_election.cc:288] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:44.978186 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a443c0d35741b9966b32a7808a97e2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:44.978717 13057 leader_election.cc:341] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b5a443c0d35741b9966b32a7808a97e2
W20240506 19:56:44.979102 13058 leader_election.cc:341] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:44.979365 13058 leader_election.cc:302] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:44.979815 13878 raft_consensus.cc:2726] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:44.979976 13245 tablet_bootstrap.cc:654] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:44.995461 13249 tablet_bootstrap.cc:492] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:44.995656 13249 ts_tablet_manager.cc:1367] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20240506 19:56:44.996589 13249 raft_consensus.cc:348] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.996910 13249 raft_consensus.cc:374] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.997009 13249 raft_consensus.cc:729] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:44.998230 13244 tablet_bootstrap.cc:492] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:44.998245 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e] alter complete (version 2)
I20240506 19:56:44.998440 13244 ts_tablet_manager.cc:1367] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.000s
I20240506 19:56:44.999208 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b2476ed5d80041e68f12fa12ddef7008"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:44.999320 13244 raft_consensus.cc:348] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:44.999558 13244 raft_consensus.cc:374] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:44.999634 13244 raft_consensus.cc:729] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:44.999948 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2476ed5d80041e68f12fa12ddef7008 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:44.999934 13244 consensus_queue.cc:260] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:45.001315 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efc41afd84a74652a82cc639bf562d99 has no consensus state
I20240506 19:56:45.001675 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b8e4c400be8f434295e3730c95d14641" } new_table_name: "default.Table_e3ab58eefbbc4d9fac784342d0775037" modify_external_catalogs: true
I20240506 19:56:45.003291 13244 ts_tablet_manager.cc:1398] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:45.003710 13244 tablet_bootstrap.cc:492] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.003719 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_df0ba23556a64063af810291bf81830e" } modify_external_catalogs: true
I20240506 19:56:45.003917 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_df0ba23556a64063af810291bf81830e" } modify_external_catalogs: true
I20240506 19:56:45.005784 13244 tablet_bootstrap.cc:654] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.006784 12520 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:45.007683 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5c4a9e274f4747b7aad740d422b75e23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.008293 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c4a9e274f4747b7aad740d422b75e23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.005637 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:44.997288 13249 consensus_queue.cc:260] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.025240 13249 ts_tablet_manager.cc:1398] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240506 19:56:45.026577 13249 tablet_bootstrap.cc:492] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:45.026795 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fd12a9cc3924f3eb8aec502e9fa2d47 consensus state has no leader
I20240506 19:56:45.028153 13244 tablet_bootstrap.cc:492] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:45.028426 13244 ts_tablet_manager.cc:1367] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20240506 19:56:45.030843 13244 raft_consensus.cc:348] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.031172 13244 raft_consensus.cc:374] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.031339 13244 raft_consensus.cc:729] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.031709 13244 consensus_queue.cc:260] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.035183 13244 ts_tablet_manager.cc:1398] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:45.037981 13244 tablet_bootstrap.cc:492] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.040405 13244 tablet_bootstrap.cc:654] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.050501 13249 tablet_bootstrap.cc:654] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.053457 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:45.054467 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ae36658079f748608197f569b0133127"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.055058 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae36658079f748608197f569b0133127 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.059672 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet da06187c83b94d03ae4ebb67a2873cf0 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:45.050839 13242 ts_tablet_manager.cc:1888] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.060076 13242 log.cc:1196] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/393073bca2a04d15b3e583700262f3e0
I20240506 19:56:45.060546 13242 ts_tablet_manager.cc:1909] T 393073bca2a04d15b3e583700262f3e0 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:45.060617 13124 raft_consensus.cc:2454] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:45.061221 12922 leader_election.cc:302] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.067946 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5ae6e15691764b5f979643fff861ff95" } new_table_name: "default.Table_be0171ec0e714233b40b175da1e82b4d" modify_external_catalogs: true
I20240506 19:56:45.069691 13864 raft_consensus.cc:2781] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:45.069903 13864 raft_consensus.cc:686] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:45.069907 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b41d42bb850b4a5b9b99fcacd93be3fb" } new_table_name: "default.Table_b7e1ec0e7d9645cc9cf705d1ed10fd5d" modify_external_catalogs: true
I20240506 19:56:45.070176 13864 consensus_queue.cc:237] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.074807 12814 tablet_service.cc:1467] Processing CreateTablet for tablet e6b24176d9d941d4a9dfb97e24b91ab6 (DEFAULT_TABLE table=default.Table_330c25efd3544e359efe8b4e731ce71c [id=ee35b19075f04bfea6c69caa44bc5ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.075301 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b24176d9d941d4a9dfb97e24b91ab6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:45.077926 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 673dc7c83f8748a09c41d51b42214aaf consensus state has no leader
I20240506 19:56:45.079982 12948 tablet_service.cc:1467] Processing CreateTablet for tablet bc31124f917c4e73bf910a9e7b7b443d (DEFAULT_TABLE table=default.Table_1698ae51beba4bd0a7c376126987543b [id=049f48b0d93a41ca93371a76316f7e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.080338 12964 tablet_service.cc:1467] Processing CreateTablet for tablet e6b24176d9d941d4a9dfb97e24b91ab6 (DEFAULT_TABLE table=default.Table_330c25efd3544e359efe8b4e731ce71c [id=ee35b19075f04bfea6c69caa44bc5ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.080652 12952 tablet_service.cc:1467] Processing CreateTablet for tablet e8a6ca6ea4fe400abbde9ec7f07cc79e (DEFAULT_TABLE table=default.Table_c895888dd3b4499a990be1065b5cdbc8 [id=6d74597bfc464c69a263031d78500eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.081180 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 2b975fdb35324fcaa11e5f573fc68b41 (DEFAULT_TABLE table=default.Table_fb159d58ea054e2883302a7bc6ea228b [id=7bdb2c56b7df4c8f89d4af97bc7b402b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.081578 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b975fdb35324fcaa11e5f573fc68b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.083262 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 78f503285b7348faabb5ea699787f088 (DEFAULT_TABLE table=default.Table_114d9e27c6d74425802c46e8610d3343 [id=efe4c73c778d4128b0ff1a51a985f5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.083803 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f503285b7348faabb5ea699787f088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.085608 12817 tablet_service.cc:1467] Processing CreateTablet for tablet bc31124f917c4e73bf910a9e7b7b443d (DEFAULT_TABLE table=default.Table_1698ae51beba4bd0a7c376126987543b [id=049f48b0d93a41ca93371a76316f7e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.085701 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 65315d8ac5ff408bae8593681335e5b1 (DEFAULT_TABLE table=default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.086000 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc31124f917c4e73bf910a9e7b7b443d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.080652 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 78f503285b7348faabb5ea699787f088 (DEFAULT_TABLE table=default.Table_114d9e27c6d74425802c46e8610d3343 [id=efe4c73c778d4128b0ff1a51a985f5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.087836 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:45.088145 12514 catalog_manager.cc:5484] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:45.088627 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]). Expected version 3 got 0
I20240506 19:56:45.089222 12809 tablet_service.cc:1467] Processing CreateTablet for tablet e8a6ca6ea4fe400abbde9ec7f07cc79e (DEFAULT_TABLE table=default.Table_c895888dd3b4499a990be1065b5cdbc8 [id=6d74597bfc464c69a263031d78500eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.089736 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a6ca6ea4fe400abbde9ec7f07cc79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.090556 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 2b975fdb35324fcaa11e5f573fc68b41 (DEFAULT_TABLE table=default.Table_fb159d58ea054e2883302a7bc6ea228b [id=7bdb2c56b7df4c8f89d4af97bc7b402b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.091637 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 65315d8ac5ff408bae8593681335e5b1 (DEFAULT_TABLE table=default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.091846 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet f8ffb88e0a0c4125b04f4d8d328da3f5 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:45.092677 13089 tablet_service.cc:1467] Processing CreateTablet for tablet e6b24176d9d941d4a9dfb97e24b91ab6 (DEFAULT_TABLE table=default.Table_330c25efd3544e359efe8b4e731ce71c [id=ee35b19075f04bfea6c69caa44bc5ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.093230 13093 tablet_service.cc:1467] Processing CreateTablet for tablet e8a6ca6ea4fe400abbde9ec7f07cc79e (DEFAULT_TABLE table=default.Table_c895888dd3b4499a990be1065b5cdbc8 [id=6d74597bfc464c69a263031d78500eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.092694 13878 raft_consensus.cc:482] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.093502 13878 raft_consensus.cc:504] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.093699 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 65315d8ac5ff408bae8593681335e5b1 (DEFAULT_TABLE table=default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.093802 13092 tablet_service.cc:1467] Processing CreateTablet for tablet bc31124f917c4e73bf910a9e7b7b443d (DEFAULT_TABLE table=default.Table_1698ae51beba4bd0a7c376126987543b [id=049f48b0d93a41ca93371a76316f7e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.094293 13878 leader_election.cc:288] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:45.094905 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 398) took 4031 ms. Trace:
I20240506 19:56:45.095014 13242 rpcz_store.cc:276] 0506 19:56:41.063606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.063991 (+ 385us) service_pool.cc:225] Handling call
0506 19:56:44.907376 (+3843385us) ts_tablet_manager.cc:1120] Deleting tablet 393073bca2a04d15b3e583700262f3e0
0506 19:56:44.907387 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:44.969433 (+ 62046us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.050680 (+ 81247us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.094833 (+ 44153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827831}
I20240506 19:56:45.095603 13242 tablet_replica.cc:330] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:45.095927 13242 raft_consensus.cc:2229] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:45.096352 13242 pending_rounds.cc:62] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Trying to abort 1 pending ops.
I20240506 19:56:45.096423 13242 pending_rounds.cc:68] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744048252977152 op_type: NO_OP noop_request { }
I20240506 19:56:45.096568 13242 raft_consensus.cc:2866] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:45.096683 13242 raft_consensus.cc:2258] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.093191 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 2b975fdb35324fcaa11e5f573fc68b41 (DEFAULT_TABLE table=default.Table_fb159d58ea054e2883302a7bc6ea228b [id=7bdb2c56b7df4c8f89d4af97bc7b402b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.098399 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 78f503285b7348faabb5ea699787f088 (DEFAULT_TABLE table=default.Table_114d9e27c6d74425802c46e8610d3343 [id=efe4c73c778d4128b0ff1a51a985f5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.098680 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8335f9308046279a25a4a95d8e0f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.101473 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 42151ea2c1214c149d7dcac9a2c447aa (DEFAULT_TABLE table=default.Table_cda4f793e543441f94c07714c44ad670 [id=109c3b8f7eea4aa1a8eca013bd7239f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.102898 13243 ts_tablet_manager.cc:1888] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.103085 13243 log.cc:1196] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f3e7329b22414da886dee4e1ec6bf3ad
I20240506 19:56:45.103741 13243 ts_tablet_manager.cc:1909] T f3e7329b22414da886dee4e1ec6bf3ad P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:45.105079 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65315d8ac5ff408bae8593681335e5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.105665 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b24176d9d941d4a9dfb97e24b91ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.105890 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 42151ea2c1214c149d7dcac9a2c447aa (DEFAULT_TABLE table=default.Table_cda4f793e543441f94c07714c44ad670 [id=109c3b8f7eea4aa1a8eca013bd7239f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.105899 12961 tablet_service.cc:1467] Processing CreateTablet for tablet fdef0a40f2e346a798b01e83a960de09 (DEFAULT_TABLE table=default.Table_54e2ae0fb2324776911994a064b540f0 [id=58db608d6c1a4044875b4363560d781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.106262 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f503285b7348faabb5ea699787f088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.107115 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdef0a40f2e346a798b01e83a960de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.108043 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a6ca6ea4fe400abbde9ec7f07cc79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.108592 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42151ea2c1214c149d7dcac9a2c447aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.109092 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc31124f917c4e73bf910a9e7b7b443d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:45.110603 13057 leader_election.cc:341] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 0e2005ef58fd4fcea16abe771ee385f8
W20240506 19:56:45.111987 12507 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 838d9e7986164ffd916d2d842dc61f6e (table default.Table_b41d42bb850b4a5b9b99fcacd93be3fb [id=f424bbb43b2a47449f998c21dae05cb9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 838d9e7986164ffd916d2d842dc61f6e consensus state has no leader
I20240506 19:56:45.111945 13086 tablet_service.cc:1467] Processing CreateTablet for tablet fdef0a40f2e346a798b01e83a960de09 (DEFAULT_TABLE table=default.Table_54e2ae0fb2324776911994a064b540f0 [id=58db608d6c1a4044875b4363560d781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.113510 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 393073bca2a04d15b3e583700262f3e0 (table default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c [id=fd3051399ebf4aa2b31bbd709b120d48]) successfully deleted
I20240506 19:56:45.118754 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2005ef58fd4fcea16abe771ee385f8" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:45.119433 13058 leader_election.cc:341] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.119603 13058 leader_election.cc:302] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.120132 13878 raft_consensus.cc:2726] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.122193 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 42151ea2c1214c149d7dcac9a2c447aa (DEFAULT_TABLE table=default.Table_cda4f793e543441f94c07714c44ad670 [id=109c3b8f7eea4aa1a8eca013bd7239f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.122821 12824 tablet_service.cc:1467] Processing CreateTablet for tablet fdef0a40f2e346a798b01e83a960de09 (DEFAULT_TABLE table=default.Table_54e2ae0fb2324776911994a064b540f0 [id=58db608d6c1a4044875b4363560d781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.129212 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b975fdb35324fcaa11e5f573fc68b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.129590 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65315d8ac5ff408bae8593681335e5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.130021 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdef0a40f2e346a798b01e83a960de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.133212 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 449eeb67b602425bb40caa69b6a78a5a (DEFAULT_TABLE table=default.Table_3cd77b3dbc674e2099537f5b46662b86 [id=f38b135c9bd24ddcac7c2d6b3f05a1d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.133626 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42151ea2c1214c149d7dcac9a2c447aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.133816 13242 ts_tablet_manager.cc:1875] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.136685 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 (DEFAULT_TABLE table=default.Table_8b6089202064458fa076b29994bd285e [id=fadd58c3f23f4fd79188f29e0c5fc5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.137145 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 (DEFAULT_TABLE table=default.Table_8b6089202064458fa076b29994bd285e [id=fadd58c3f23f4fd79188f29e0c5fc5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.137391 13095 tablet_service.cc:1467] Processing CreateTablet for tablet ea202c26cd154d19b6926ea28bd61844 (DEFAULT_TABLE table=default.Table_dfe68f19a2874be5bdd3c0321ba9872d [id=f22bf1138894495291b1ac38b34d0e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.137557 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46fd9906504e4b95a5b05b01dc3ea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.138615 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_768e769dac9046d582bf4c77e1221d9d" } modify_external_catalogs: true
I20240506 19:56:45.138851 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_768e769dac9046d582bf4c77e1221d9d" } modify_external_catalogs: true
I20240506 19:56:45.141325 12812 tablet_service.cc:1467] Processing CreateTablet for tablet ea202c26cd154d19b6926ea28bd61844 (DEFAULT_TABLE table=default.Table_dfe68f19a2874be5bdd3c0321ba9872d [id=f22bf1138894495291b1ac38b34d0e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.142068 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea202c26cd154d19b6926ea28bd61844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.144116 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ff179844ac74496183bb76ddcabbcc75" } modify_external_catalogs: true
I20240506 19:56:45.144330 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ff179844ac74496183bb76ddcabbcc75" } modify_external_catalogs: true
I20240506 19:56:45.147372 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e091d2a1ea93426dacdda3d117da60b0" } modify_external_catalogs: true
I20240506 19:56:45.147563 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e091d2a1ea93426dacdda3d117da60b0" } modify_external_catalogs: true
I20240506 19:56:45.149173 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 5988b89cb52e4f858b52862138bed025 with delete_type TABLET_DATA_DELETED (replaced by tablet 4973e600038742a6bec0bb2d8f3551fe) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.153826 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f3e7329b22414da886dee4e1ec6bf3ad (table default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc [id=49660c755f8f470b8b8dc301ca27bb26]) successfully deleted
W20240506 19:56:45.154109 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7de7397db1e74b4488a3f9a70136bccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7de7397db1e74b4488a3f9a70136bccb already in progress: creating tablet
W20240506 19:56:45.154302 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 04738def56324a368f8182543107e32e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04738def56324a368f8182543107e32e already in progress: creating tablet
W20240506 19:56:45.154455 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 already in progress: creating tablet
W20240506 19:56:45.154727 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 50ef5a2799b447f992207452dbe39645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ef5a2799b447f992207452dbe39645 already in progress: creating tablet
W20240506 19:56:45.154881 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 09cb71b5962f4561bf0fdbe12bf45296 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09cb71b5962f4561bf0fdbe12bf45296 already in progress: creating tablet
W20240506 19:56:45.155400 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 already in progress: creating tablet
I20240506 19:56:45.156000 13246 ts_tablet_manager.cc:1888] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.156366 13246 log.cc:1196] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f0a8b12f36e74c95ad6eaf30e432f3e9
I20240506 19:56:45.156256 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdef0a40f2e346a798b01e83a960de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.157033 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc31124f917c4e73bf910a9e7b7b443d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.157017 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 449eeb67b602425bb40caa69b6a78a5a (DEFAULT_TABLE table=default.Table_3cd77b3dbc674e2099537f5b46662b86 [id=f38b135c9bd24ddcac7c2d6b3f05a1d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.157794 13246 ts_tablet_manager.cc:1909] T f0a8b12f36e74c95ad6eaf30e432f3e9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:45.157984 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a6ca6ea4fe400abbde9ec7f07cc79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.158728 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65315d8ac5ff408bae8593681335e5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.158995 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46fd9906504e4b95a5b05b01dc3ea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.159397 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b975fdb35324fcaa11e5f573fc68b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.159734 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea202c26cd154d19b6926ea28bd61844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.160138 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449eeb67b602425bb40caa69b6a78a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.160641 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f503285b7348faabb5ea699787f088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.161201 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b24176d9d941d4a9dfb97e24b91ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.161746 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42151ea2c1214c149d7dcac9a2c447aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.169464 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449eeb67b602425bb40caa69b6a78a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.216959 12955 tablet_service.cc:1467] Processing CreateTablet for tablet ea202c26cd154d19b6926ea28bd61844 (DEFAULT_TABLE table=default.Table_dfe68f19a2874be5bdd3c0321ba9872d [id=f22bf1138894495291b1ac38b34d0e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.217859 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 572) took 1013 ms. Trace:
I20240506 19:56:45.218850 13246 rpcz_store.cc:276] 0506 19:56:44.197642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.209272 (+ 11630us) service_pool.cc:225] Handling call
0506 19:56:44.873018 (+663746us) ts_tablet_manager.cc:1120] Deleting tablet f0a8b12f36e74c95ad6eaf30e432f3e9
0506 19:56:44.873026 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.061239 (+188213us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.155851 (+ 94612us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.210635 (+ 54784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":663495}
I20240506 19:56:45.219930 13246 tablet_replica.cc:330] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:45.220047 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f0a8b12f36e74c95ad6eaf30e432f3e9 (table default.Table_038664bd22da4e20b40b21843516298f [id=34aeae2e9b5546d1b405ce1432e3c0cb]) successfully deleted
I20240506 19:56:45.221148 13246 raft_consensus.cc:2229] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.221608 13246 raft_consensus.cc:2258] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.226018 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea202c26cd154d19b6926ea28bd61844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.232404 13246 ts_tablet_manager.cc:1875] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.238525 13244 tablet_bootstrap.cc:492] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:45.238749 13244 ts_tablet_manager.cc:1367] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.201s user 0.003s sys 0.003s
I20240506 19:56:45.240227 13244 raft_consensus.cc:348] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.240480 13244 raft_consensus.cc:374] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.240562 13244 raft_consensus.cc:729] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.240833 13244 consensus_queue.cc:260] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.242249 13244 ts_tablet_manager.cc:1398] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:45.243057 13244 tablet_bootstrap.cc:492] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.246315 13244 tablet_bootstrap.cc:654] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.250008 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 (DEFAULT_TABLE table=default.Table_8b6089202064458fa076b29994bd285e [id=fadd58c3f23f4fd79188f29e0c5fc5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.250738 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46fd9906504e4b95a5b05b01dc3ea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.263917 13244 tablet_bootstrap.cc:492] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:45.264328 13244 ts_tablet_manager.cc:1367] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.001s
I20240506 19:56:45.265372 13244 raft_consensus.cc:348] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.265635 13245 tablet_bootstrap.cc:492] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.265690 13244 raft_consensus.cc:374] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.265805 13244 raft_consensus.cc:729] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.265827 13245 ts_tablet_manager.cc:1367] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.323s user 0.002s sys 0.013s
I20240506 19:56:45.266088 13244 consensus_queue.cc:260] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.268185 13244 ts_tablet_manager.cc:1398] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:45.269165 13245 raft_consensus.cc:348] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.269564 13244 tablet_bootstrap.cc:492] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.269538 13245 raft_consensus.cc:374] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.271131 13245 raft_consensus.cc:729] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.271462 13245 consensus_queue.cc:260] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.274528 13245 ts_tablet_manager.cc:1398] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:45.279453 13244 tablet_bootstrap.cc:654] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.280447 13245 tablet_bootstrap.cc:492] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.284204 13245 tablet_bootstrap.cc:654] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.297480 13887 raft_consensus.cc:482] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.297756 13887 raft_consensus.cc:504] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.298789 13887 leader_election.cc:288] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:45.300256 12783 leader_election.cc:341] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2d9230ed2e9a49ecb0a5e5d520cd03aa
W20240506 19:56:45.300968 12781 leader_election.cc:341] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2d9230ed2e9a49ecb0a5e5d520cd03aa
I20240506 19:56:45.301164 12781 leader_election.cc:302] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:45.302279 13887 raft_consensus.cc:2726] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.309253 13864 raft_consensus.cc:482] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.309476 13864 raft_consensus.cc:504] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.310233 13864 leader_election.cc:288] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.312086 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be929f0e72e4d22a9af2fe6b3072bbe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:45.312994 12922 leader_election.cc:341] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.314028 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be929f0e72e4d22a9af2fe6b3072bbe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:45.314975 12921 leader_election.cc:341] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.315101 12921 leader_election.cc:302] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.315701 13911 raft_consensus.cc:2726] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.327843 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 449eeb67b602425bb40caa69b6a78a5a (DEFAULT_TABLE table=default.Table_3cd77b3dbc674e2099537f5b46662b86 [id=f38b135c9bd24ddcac7c2d6b3f05a1d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.328346 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449eeb67b602425bb40caa69b6a78a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.331288 13243 tablet_replica.cc:330] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:45.331473 13243 raft_consensus.cc:2229] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:45.331727 13243 pending_rounds.cc:62] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:45.331794 13243 pending_rounds.cc:68] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744059170549760 op_type: NO_OP noop_request { }
I20240506 19:56:45.331895 13243 raft_consensus.cc:2866] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:45.331977 13243 raft_consensus.cc:2258] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:45.332345 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9df3ab9dff5148f9bb335a7209b29bbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9df3ab9dff5148f9bb335a7209b29bbc already in progress: creating tablet
I20240506 19:56:45.334039 13249 tablet_bootstrap.cc:492] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:45.334164 13249 ts_tablet_manager.cc:1367] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.308s user 0.048s sys 0.031s
I20240506 19:56:45.335137 13249 raft_consensus.cc:348] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.335311 13249 raft_consensus.cc:374] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.335361 13249 raft_consensus.cc:729] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.335937 13243 ts_tablet_manager.cc:1875] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.340167 13249 consensus_queue.cc:260] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.341109 13249 ts_tablet_manager.cc:1398] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:45.341467 13249 tablet_bootstrap.cc:492] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:45.343855 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 1667a0bbd3af4d678434b068c47cc518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.348668 13249 tablet_bootstrap.cc:654] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.357249 13242 ts_tablet_manager.cc:1888] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:45.357482 13242 log.cc:1196] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2502b423bd1040539fb1a3ebbece3c1c
I20240506 19:56:45.357910 13242 ts_tablet_manager.cc:1909] T 2502b423bd1040539fb1a3ebbece3c1c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:45.361778 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 399) took 4241 ms. Trace:
I20240506 19:56:45.361944 13242 rpcz_store.cc:276] 0506 19:56:41.120749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.120815 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:45.095510 (+3974695us) ts_tablet_manager.cc:1120] Deleting tablet 2502b423bd1040539fb1a3ebbece3c1c
0506 19:56:45.095519 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.253381 (+157862us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.357087 (+103706us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.361708 (+ 4621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3746733}
I20240506 19:56:45.362442 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 408) took 3982 ms. Trace:
I20240506 19:56:45.362541 13242 rpcz_store.cc:276] 0506 19:56:41.380033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.380104 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:45.362250 (+3982146us) ts_tablet_manager.cc:1120] Deleting tablet a1160ab972124630a9be7abeefb4aa56
0506 19:56:45.362261 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.362383 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3967131}
I20240506 19:56:45.362955 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 407) took 3989 ms. Trace:
I20240506 19:56:45.363040 13242 rpcz_store.cc:276] 0506 19:56:41.378612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.378723 (+ 111us) service_pool.cc:225] Handling call
0506 19:56:45.362764 (+3984041us) ts_tablet_manager.cc:1120] Deleting tablet d6e54543d1544b1c8cff3e890e787337
0506 19:56:45.362772 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.362904 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3965628}
I20240506 19:56:45.363408 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 409) took 3959 ms. Trace:
I20240506 19:56:45.363492 13242 rpcz_store.cc:276] 0506 19:56:41.403577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.403628 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:45.363260 (+3959632us) ts_tablet_manager.cc:1120] Deleting tablet 8d09eafa82a0423a8358958e78f728e2
0506 19:56:45.363268 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.363354 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3956108}
I20240506 19:56:45.363885 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 412) took 3950 ms. Trace:
I20240506 19:56:45.363979 13242 rpcz_store.cc:276] 0506 19:56:41.413424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.413700 (+ 276us) service_pool.cc:225] Handling call
0506 19:56:45.363710 (+3950010us) ts_tablet_manager.cc:1120] Deleting tablet a9e90b93023e40b3bafbddf7bbcc10b2
0506 19:56:45.363719 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.363822 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3949275}
I20240506 19:56:45.364303 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 410) took 3956 ms. Trace:
I20240506 19:56:45.364384 13242 rpcz_store.cc:276] 0506 19:56:41.407465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.407833 (+ 368us) service_pool.cc:225] Handling call
0506 19:56:45.364174 (+3956341us) ts_tablet_manager.cc:1120] Deleting tablet 234e4f3439e84b5fbb2f7bab1dbd0822
0506 19:56:45.364182 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.364251 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3949569}
I20240506 19:56:45.364733 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 411) took 3952 ms. Trace:
I20240506 19:56:45.364847 13242 rpcz_store.cc:276] 0506 19:56:41.412072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.412270 (+ 198us) service_pool.cc:225] Handling call
0506 19:56:45.364584 (+3952314us) ts_tablet_manager.cc:1120] Deleting tablet 2a7c61244620487bbbac26dcdbf8a38c
0506 19:56:45.364591 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.364678 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3947441}
I20240506 19:56:45.365180 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 430) took 3782 ms. Trace:
I20240506 19:56:45.366429 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2502b423bd1040539fb1a3ebbece3c1c (table default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]) successfully deleted
W20240506 19:56:45.366775 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a1160ab972124630a9be7abeefb4aa56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1160ab972124630a9be7abeefb4aa56 already in progress: creating tablet
W20240506 19:56:45.367076 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d6e54543d1544b1c8cff3e890e787337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6e54543d1544b1c8cff3e890e787337 already in progress: creating tablet
W20240506 19:56:45.367290 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8d09eafa82a0423a8358958e78f728e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d09eafa82a0423a8358958e78f728e2 already in progress: creating tablet
W20240506 19:56:45.367520 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a9e90b93023e40b3bafbddf7bbcc10b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e90b93023e40b3bafbddf7bbcc10b2 already in progress: creating tablet
W20240506 19:56:45.367708 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 234e4f3439e84b5fbb2f7bab1dbd0822 already in progress: creating tablet
W20240506 19:56:45.367962 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2a7c61244620487bbbac26dcdbf8a38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7c61244620487bbbac26dcdbf8a38c already in progress: creating tablet
I20240506 19:56:45.365255 13242 rpcz_store.cc:276] 0506 19:56:41.582776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.583502 (+ 726us) service_pool.cc:225] Handling call
0506 19:56:45.365047 (+3781545us) ts_tablet_manager.cc:1120] Deleting tablet 458314e4653748e48cf646c72373fc3e
0506 19:56:45.365055 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.365133 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3756723}
W20240506 19:56:45.368996 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 458314e4653748e48cf646c72373fc3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 458314e4653748e48cf646c72373fc3e already in progress: creating tablet
I20240506 19:56:45.370047 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 431) took 3786 ms. Trace:
I20240506 19:56:45.370191 13242 rpcz_store.cc:276] 0506 19:56:41.582890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.608360 (+ 25470us) service_pool.cc:225] Handling call
0506 19:56:45.369329 (+3760969us) ts_tablet_manager.cc:1120] Deleting tablet ff3049a9fa8049e5b84230e5e6e8245c
0506 19:56:45.369340 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.369442 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3760759}
I20240506 19:56:45.370316 13249 tablet_bootstrap.cc:492] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:45.370466 13249 ts_tablet_manager.cc:1367] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.004s
W20240506 19:56:45.370775 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ff3049a9fa8049e5b84230e5e6e8245c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3049a9fa8049e5b84230e5e6e8245c already in progress: creating tablet
I20240506 19:56:45.371122 13245 tablet_bootstrap.cc:492] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.371227 13249 raft_consensus.cc:348] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.371405 13242 tablet_replica.cc:330] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:45.371446 13249 raft_consensus.cc:374] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.371379 13245 ts_tablet_manager.cc:1367] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.001s
I20240506 19:56:45.371501 13249 raft_consensus.cc:729] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.371711 13242 raft_consensus.cc:2229] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.371716 13249 consensus_queue.cc:260] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.372037 13242 raft_consensus.cc:2258] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.373490 13242 ts_tablet_manager.cc:1875] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.377730 13245 raft_consensus.cc:348] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.379112 13245 raft_consensus.cc:374] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.379526 13245 raft_consensus.cc:729] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.379828 13887 raft_consensus.cc:482] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.379992 13887 raft_consensus.cc:504] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.380126 13244 tablet_bootstrap.cc:492] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:45.380316 13244 ts_tablet_manager.cc:1367] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.111s user 0.015s sys 0.018s
I20240506 19:56:45.380512 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_94aeb54aa38948f7b903f8074dba8b52" } modify_external_catalogs: true
I20240506 19:56:45.380764 13887 leader_election.cc:288] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.380733 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_94aeb54aa38948f7b903f8074dba8b52" } modify_external_catalogs: true
I20240506 19:56:45.381106 13245 consensus_queue.cc:260] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.381189 13244 raft_consensus.cc:348] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.381417 13244 raft_consensus.cc:374] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.381522 13244 raft_consensus.cc:729] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:45.381667 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 260d7ab458794df496012cb80466f59c consensus state has no leader
I20240506 19:56:45.381783 13244 consensus_queue.cc:260] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.382184 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c1ceccbbb3344105a1314370d915f2cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.382823 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1ceccbbb3344105a1314370d915f2cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.383339 13245 ts_tablet_manager.cc:1398] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240506 19:56:45.383832 13244 ts_tablet_manager.cc:1398] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240506 19:56:45.384218 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 838d9e7986164ffd916d2d842dc61f6e consensus state has no leader
I20240506 19:56:45.385412 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:45.386251 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:56:45.386209 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7ae101bd10f242f28c19bd685449fce8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.386816 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ae101bd10f242f28c19bd685449fce8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.387068 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f68f57002f8144caaaf37c2838f98e1f
I20240506 19:56:45.387841 13244 tablet_bootstrap.cc:492] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.387817 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a62a1c7b556341a88bc49f8caa3c8238" } new_table_name: "default.Table_e2c60bcc57d34618a8cc33cab56b7a78" modify_external_catalogs: true
I20240506 19:56:45.389709 13245 tablet_bootstrap.cc:492] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.389927 13243 ts_tablet_manager.cc:1888] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:45.390061 13243 log.cc:1196] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/00de09ddcdd948a6a3500a6c9d814c14
W20240506 19:56:45.390226 12781 leader_election.cc:341] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:45.390452 13243 ts_tablet_manager.cc:1909] T 00de09ddcdd948a6a3500a6c9d814c14 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:45.391130 12783 leader_election.cc:341] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:45.391244 12783 leader_election.cc:302] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:45.392001 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f57002f8144caaaf37c2838f98e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.392261 13871 raft_consensus.cc:2726] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.393203 13244 tablet_bootstrap.cc:654] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.397213 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ebd7b994584b2bab6d27f493deab92) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.400442 13245 tablet_bootstrap.cc:654] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.401065 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f57002f8144caaaf37c2838f98e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.401224 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f57002f8144caaaf37c2838f98e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.403126 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ebd7b994584b2bab6d27f493deab92) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.404525 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 99f4861a01404663870ce91978c8af4f
I20240506 19:56:45.406313 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.406636 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.408001 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c656e16c1045dea79bc23b64fdd5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.408572 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:45.409430 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 55d9f32618b64ffa92b5a868081c36a6
I20240506 19:56:45.410210 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d9f32618b64ffa92b5a868081c36a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.410274 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.410715 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d9f32618b64ffa92b5a868081c36a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.410256 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a5d3e2fa72f2440eae3408f5b092bd7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.411691 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5d3e2fa72f2440eae3408f5b092bd7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.414330 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d9f32618b64ffa92b5a868081c36a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.415693 13245 tablet_bootstrap.cc:492] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.415854 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.415902 13245 ts_tablet_manager.cc:1367] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.001s
I20240506 19:56:45.416921 13245 raft_consensus.cc:348] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.417203 13245 raft_consensus.cc:374] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.417351 13245 raft_consensus.cc:729] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.417613 13245 consensus_queue.cc:260] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.419874 13245 ts_tablet_manager.cc:1398] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:45.420363 13245 tablet_bootstrap.cc:492] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.420539 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ac829ce2695e44849b1d44918904dc8a" } new_table_name: "default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17" modify_external_catalogs: true
I20240506 19:56:45.386811 12519 catalog_manager.cc:6554] ReplaceTablet: tablet da06187c83b94d03ae4ebb67a2873cf0 deleted and replaced by tablet d1ebd7b994584b2bab6d27f493deab92
I20240506 19:56:45.423835 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e1bc856560494ef0ac2a3d693b45a7c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.424441 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1bc856560494ef0ac2a3d693b45a7c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.425045 13245 tablet_bootstrap.cc:654] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.426875 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe with delete_type TABLET_DATA_DELETED (replaced by tablet 2f7f7171d06948388bc0fdb3514202ed) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.427385 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_460654e013224b0b862d4c4663839cc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.427927 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_460654e013224b0b862d4c4663839cc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.429929 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet cd57563cc6c140c184d1725a2da6a475
I20240506 19:56:45.430783 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3b6103041f344349ac34c8f87e87adae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.430908 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet cd57563cc6c140c184d1725a2da6a475 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.431075 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet cd57563cc6c140c184d1725a2da6a475 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:45.431332 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b6103041f344349ac34c8f87e87adae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.434623 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet cd57563cc6c140c184d1725a2da6a475 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.435559 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ebd7b994584b2bab6d27f493deab92) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.440846 13245 tablet_bootstrap.cc:492] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.440742 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a40f3b8780d740ee995e313e40e3b6fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.441251 13245 ts_tablet_manager.cc:1367] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20240506 19:56:45.442926 13245 raft_consensus.cc:348] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.443286 13245 raft_consensus.cc:374] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.443400 13245 raft_consensus.cc:729] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.443686 13245 consensus_queue.cc:260] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:45.445725 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a40f3b8780d740ee995e313e40e3b6fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.446249 13245 ts_tablet_manager.cc:1398] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:45.446832 13245 tablet_bootstrap.cc:492] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.450327 13249 ts_tablet_manager.cc:1398] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.080s user 0.000s sys 0.002s
I20240506 19:56:45.451844 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 00de09ddcdd948a6a3500a6c9d814c14 (table default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]) successfully deleted
W20240506 19:56:45.452929 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f68f57002f8144caaaf37c2838f98e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f68f57002f8144caaaf37c2838f98e1f already in progress: creating tablet
W20240506 19:56:45.453145 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 55d9f32618b64ffa92b5a868081c36a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55d9f32618b64ffa92b5a868081c36a6 already in progress: creating tablet
W20240506 19:56:45.453454 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 99f4861a01404663870ce91978c8af4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99f4861a01404663870ce91978c8af4f already in progress: creating tablet
I20240506 19:56:45.451378 13243 tablet_replica.cc:330] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:45.456655 13245 tablet_bootstrap.cc:654] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.456940 13243 raft_consensus.cc:2229] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.457191 13243 raft_consensus.cc:2258] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.457911 13243 ts_tablet_manager.cc:1875] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.459079 13246 ts_tablet_manager.cc:1888] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:45.459244 13246 log.cc:1196] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/acd3fc62e9614cf3bde0202ee6832bb3
I20240506 19:56:45.459674 13246 ts_tablet_manager.cc:1909] T acd3fc62e9614cf3bde0202ee6832bb3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:45.465612 13249 tablet_bootstrap.cc:492] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:45.466521 13242 ts_tablet_manager.cc:1888] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.466704 13242 log.cc:1196] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ee59a76d52fc48d7bcf3256c359d3c02
I20240506 19:56:45.467182 13242 ts_tablet_manager.cc:1909] T ee59a76d52fc48d7bcf3256c359d3c02 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:45.472488 13249 tablet_bootstrap.cc:654] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.477589 13245 tablet_bootstrap.cc:492] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.477785 13245 ts_tablet_manager.cc:1367] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.002s
I20240506 19:56:45.478624 13245 raft_consensus.cc:348] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.478843 13245 raft_consensus.cc:374] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.478928 13245 raft_consensus.cc:729] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.479203 13245 consensus_queue.cc:260] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.479876 13245 ts_tablet_manager.cc:1398] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:45.480278 13245 tablet_bootstrap.cc:492] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:45.481654 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b625c2bbd89746bc90b8c4c2483b3741 consensus state has no leader
I20240506 19:56:45.482038 13245 tablet_bootstrap.cc:654] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.483429 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ff9dd351a2594431addcdc137481cd02" } new_table_name: "default.Table_8b082b3b3998450aac6c69f4dfd84585" modify_external_catalogs: true
W20240506 19:56:45.484933 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6352d2c630204df3a24e299e167f92d4 consensus state has no leader
I20240506 19:56:45.486654 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet 9944e5bfe0a741d69b8f9b6a9dad84dd from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:45.487249 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c19567619b64ee0a8dd637d7f69ce3e from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:45.488528 13249 tablet_bootstrap.cc:492] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:45.488689 13249 ts_tablet_manager.cc:1367] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240506 19:56:45.488668 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 576) took 1102 ms. Trace:
I20240506 19:56:45.488776 13246 rpcz_store.cc:276] 0506 19:56:44.386597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.387162 (+ 565us) service_pool.cc:225] Handling call
0506 19:56:45.219842 (+832680us) ts_tablet_manager.cc:1120] Deleting tablet acd3fc62e9614cf3bde0202ee6832bb3
0506 19:56:45.219856 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.360115 (+140259us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.458925 (+ 98810us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.488583 (+ 29658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":832373}
I20240506 19:56:45.489456 13249 raft_consensus.cc:348] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.489650 13249 raft_consensus.cc:374] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.489717 13249 raft_consensus.cc:729] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.489969 13249 consensus_queue.cc:260] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.490648 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 2502b423bd1040539fb1a3ebbece3c1c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:45.491122 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 65da8b15e0c6496bb86b47c49e348af0 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:45.494022 13249 ts_tablet_manager.cc:1398] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:45.495103 13887 raft_consensus.cc:482] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 059a5c4a3f7e49ef95af4b8341702745)
I20240506 19:56:45.495273 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_79eff2000c124513b48aeefee23bb9a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.495393 13887 raft_consensus.cc:504] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.496340 13887 leader_election.cc:288] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.496572 13249 tablet_bootstrap.cc:492] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:45.498082 13249 tablet_bootstrap.cc:654] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:45.498047 12783 leader_election.cc:341] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 9c18888ae3d6420ca1de3b13fc36cf6c
W20240506 19:56:45.499172 12781 leader_election.cc:341] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 9c18888ae3d6420ca1de3b13fc36cf6c
I20240506 19:56:45.499429 12781 leader_election.cc:302] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:45.505992 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]) successfully deleted
I20240506 19:56:45.506700 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.507107 13887 raft_consensus.cc:2726] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:45.507843 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ea7e073296ce46e786362aaed96eb4dd
I20240506 19:56:45.508996 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7e073296ce46e786362aaed96eb4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.509227 13887 raft_consensus.cc:482] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.509426 13887 raft_consensus.cc:504] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.510090 13249 tablet_bootstrap.cc:492] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:45.510234 13249 ts_tablet_manager.cc:1367] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240506 19:56:45.510274 13887 leader_election.cc:288] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.511008 13249 raft_consensus.cc:348] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.511212 13249 raft_consensus.cc:374] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.511271 13249 raft_consensus.cc:729] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.511518 13249 consensus_queue.cc:260] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.512384 13249 ts_tablet_manager.cc:1398] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:45.512681 13249 tablet_bootstrap.cc:492] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:45.513859 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7e073296ce46e786362aaed96eb4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.514600 13249 tablet_bootstrap.cc:654] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:45.515606 12781 leader_election.cc:341] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 1667a0bbd3af4d678434b068c47cc518
W20240506 19:56:45.516095 12783 leader_election.cc:341] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:45.516266 12783 leader_election.cc:302] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:45.516796 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 582) took 1059 ms. Trace:
I20240506 19:56:45.516947 13887 raft_consensus.cc:2726] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.516906 13246 rpcz_store.cc:276] 0506 19:56:44.456886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.457456 (+ 570us) service_pool.cc:225] Handling call
0506 19:56:45.489145 (+1031689us) ts_tablet_manager.cc:1120] Deleting tablet 0aa8f9aaca444c839a88c07f58cb6ad6
0506 19:56:45.489165 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.516714 (+ 27549us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000957}
I20240506 19:56:45.517274 13246 tablet_replica.cc:330] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:45.517493 13246 raft_consensus.cc:2229] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.517681 13246 raft_consensus.cc:2258] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.518260 13246 ts_tablet_manager.cc:1875] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:45.523093 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79eff2000c124513b48aeefee23bb9a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:45.524868 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aa8f9aaca444c839a88c07f58cb6ad6 already in progress: creating tablet
I20240506 19:56:45.525642 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7e073296ce46e786362aaed96eb4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.528766 13249 tablet_bootstrap.cc:492] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:45.528932 13249 ts_tablet_manager.cc:1367] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240506 19:56:45.529793 13249 raft_consensus.cc:348] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.529999 13249 raft_consensus.cc:374] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.530054 13249 raft_consensus.cc:729] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.530293 13249 consensus_queue.cc:260] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.531234 13249 ts_tablet_manager.cc:1398] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:45.531533 13249 tablet_bootstrap.cc:492] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:45.532094 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe with delete_type TABLET_DATA_DELETED (replaced by tablet 2f7f7171d06948388bc0fdb3514202ed) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.532383 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet db13b58d17c94e0f82052f823915d9af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.533483 13249 tablet_bootstrap.cc:654] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.538717 13244 tablet_bootstrap.cc:492] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:45.538954 13244 ts_tablet_manager.cc:1367] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.151s user 0.013s sys 0.027s
I20240506 19:56:45.539966 13244 raft_consensus.cc:348] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.540284 13244 raft_consensus.cc:374] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.540403 13244 raft_consensus.cc:729] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.540674 13244 consensus_queue.cc:260] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.541723 13244 ts_tablet_manager.cc:1398] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240506 19:56:45.542306 13244 tablet_bootstrap.cc:492] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.543507 13864 raft_consensus.cc:482] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.543674 13864 raft_consensus.cc:504] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.544375 13244 tablet_bootstrap.cc:654] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.544358 13864 leader_election.cc:288] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.545320 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:45.545571 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
I20240506 19:56:45.545868 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:45.547535 12921 leader_election.cc:341] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.547585 13249 tablet_bootstrap.cc:492] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:45.548583 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6788948824bc43638d1e10322195e7e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.549124 13249 ts_tablet_manager.cc:1367] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
W20240506 19:56:45.549259 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6788948824bc43638d1e10322195e7e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:45.551613 12922 leader_election.cc:341] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.551748 12922 leader_election.cc:302] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.552211 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 432) took 3969 ms. Trace:
I20240506 19:56:45.552345 13242 rpcz_store.cc:276] 0506 19:56:41.583046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.608558 (+ 25512us) service_pool.cc:225] Handling call
0506 19:56:45.371062 (+3762504us) ts_tablet_manager.cc:1120] Deleting tablet ee59a76d52fc48d7bcf3256c359d3c02
0506 19:56:45.371071 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.458712 (+ 87641us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.466288 (+ 7576us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.552150 (+ 85862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3761986}
I20240506 19:56:45.552517 12842 raft_consensus.cc:3037] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:45.553519 13864 raft_consensus.cc:2726] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.553958 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ee59a76d52fc48d7bcf3256c359d3c02 (table default.Table_cfd9c4e6fbe449f3a1601142d10c4cb7 [id=d9b546ebb0c146a7920cc05c08a29e32]) successfully deleted
I20240506 19:56:45.555949 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d380bf30599d4130bbb80b4f26837779 (table default.Table_fd07c6b21a494057a0f71bb60e892972 [id=873dfe33ce7141f2bc25d7575a63fc9f]). Expected version 1 got 0
I20240506 19:56:45.557545 13249 raft_consensus.cc:348] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.557760 13249 raft_consensus.cc:374] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.557816 13249 raft_consensus.cc:729] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.558044 13249 consensus_queue.cc:260] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.562389 13249 ts_tablet_manager.cc:1398] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240506 19:56:45.552834 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 429) took 3981 ms. Trace:
I20240506 19:56:45.563577 13242 rpcz_store.cc:276] 0506 19:56:41.582637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.583412 (+ 775us) service_pool.cc:225] Handling call
0506 19:56:45.552657 (+3969245us) ts_tablet_manager.cc:1120] Deleting tablet b6fa6add23b8403ba5a0a65ac6752a5b
0506 19:56:45.552666 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.552794 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3942660}
I20240506 19:56:45.564077 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 428) took 3993 ms. Trace:
I20240506 19:56:45.564206 13242 rpcz_store.cc:276] 0506 19:56:41.571024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.571830 (+ 806us) service_pool.cc:225] Handling call
0506 19:56:45.563877 (+3992047us) ts_tablet_manager.cc:1120] Deleting tablet 2417a535329341d8bb9f8218770f890d
0506 19:56:45.563888 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.564034 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3942158}
I20240506 19:56:45.564586 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 434) took 3820 ms. Trace:
I20240506 19:56:45.564690 13242 rpcz_store.cc:276] 0506 19:56:41.744174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.744327 (+ 153us) service_pool.cc:225] Handling call
0506 19:56:45.564456 (+3820129us) ts_tablet_manager.cc:1120] Deleting tablet de26793c006440dc9aecc07963ddb85e
0506 19:56:45.564463 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.564552 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3809942}
I20240506 19:56:45.565062 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 435) took 3812 ms. Trace:
I20240506 19:56:45.565179 13242 rpcz_store.cc:276] 0506 19:56:41.752660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.753692 (+ 1032us) service_pool.cc:225] Handling call
0506 19:56:45.564933 (+3811241us) ts_tablet_manager.cc:1120] Deleting tablet 9e3d13322b4a4a2db3b6d425679f68ed
0506 19:56:45.564942 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.565032 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3752432}
I20240506 19:56:45.565515 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 437) took 3699 ms. Trace:
I20240506 19:56:45.565609 13242 rpcz_store.cc:276] 0506 19:56:41.873391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:41.873672 (+ 281us) service_pool.cc:225] Handling call
0506 19:56:45.565391 (+3691719us) ts_tablet_manager.cc:1120] Deleting tablet f3d57249f30a44dab3384092ec20b515
0506 19:56:45.565398 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.565485 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3672811}
I20240506 19:56:45.565907 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 449) took 3548 ms. Trace:
I20240506 19:56:45.566013 13242 rpcz_store.cc:276] 0506 19:56:42.017852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.019582 (+ 1730us) service_pool.cc:225] Handling call
0506 19:56:45.565799 (+3546217us) ts_tablet_manager.cc:1120] Deleting tablet 0f28c1f55fd14aca86cb00e43b3a6b76
0506 19:56:45.565806 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.565877 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3533392}
I20240506 19:56:45.566347 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 448) took 3548 ms. Trace:
I20240506 19:56:45.566452 13242 rpcz_store.cc:276] 0506 19:56:42.017760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.019556 (+ 1796us) service_pool.cc:225] Handling call
0506 19:56:45.566233 (+3546677us) ts_tablet_manager.cc:1120] Deleting tablet f3a8531a1793404699532295164a3895
0506 19:56:45.566241 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.566312 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3533694}
I20240506 19:56:45.566772 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 447) took 3549 ms. Trace:
I20240506 19:56:45.566910 13242 rpcz_store.cc:276] 0506 19:56:42.017654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.019524 (+ 1870us) service_pool.cc:225] Handling call
0506 19:56:45.566642 (+3547118us) ts_tablet_manager.cc:1120] Deleting tablet bc85233b180f4b30a86bb65c86af684e
0506 19:56:45.566650 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.566737 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3532834}
I20240506 19:56:45.567217 13242 tablet_replica.cc:330] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:45.567461 13242 raft_consensus.cc:2229] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.567638 13242 raft_consensus.cc:2258] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:45.569830 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fe8d4891c484627ac729971ff7aad3e consensus state has no leader
W20240506 19:56:45.570816 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b6fa6add23b8403ba5a0a65ac6752a5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fa6add23b8403ba5a0a65ac6752a5b already in progress: creating tablet
W20240506 19:56:45.571066 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2417a535329341d8bb9f8218770f890d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2417a535329341d8bb9f8218770f890d already in progress: creating tablet
W20240506 19:56:45.571290 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet de26793c006440dc9aecc07963ddb85e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de26793c006440dc9aecc07963ddb85e already in progress: creating tablet
W20240506 19:56:45.571467 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9e3d13322b4a4a2db3b6d425679f68ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e3d13322b4a4a2db3b6d425679f68ed already in progress: creating tablet
W20240506 19:56:45.571646 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f3d57249f30a44dab3384092ec20b515 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3d57249f30a44dab3384092ec20b515 already in progress: creating tablet
W20240506 19:56:45.571842 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0f28c1f55fd14aca86cb00e43b3a6b76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f28c1f55fd14aca86cb00e43b3a6b76 already in progress: creating tablet
W20240506 19:56:45.572106 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f3a8531a1793404699532295164a3895 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a8531a1793404699532295164a3895 already in progress: creating tablet
I20240506 19:56:45.572229 13249 tablet_bootstrap.cc:492] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:45.572333 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bc85233b180f4b30a86bb65c86af684e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc85233b180f4b30a86bb65c86af684e already in progress: creating tablet
I20240506 19:56:45.573948 13244 tablet_bootstrap.cc:492] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:45.574224 13244 ts_tablet_manager.cc:1367] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.021s
I20240506 19:56:45.574746 13249 tablet_bootstrap.cc:654] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.575017 13244 raft_consensus.cc:348] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.575253 13244 raft_consensus.cc:374] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.575323 13244 raft_consensus.cc:729] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.575556 13244 consensus_queue.cc:260] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.576107 13244 ts_tablet_manager.cc:1398] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:45.576411 13244 tablet_bootstrap.cc:492] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:45.576632 12515 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
I20240506 19:56:45.578075 13244 tablet_bootstrap.cc:654] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.590510 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec5458abb234366af4c23754a54d68a (DEFAULT_TABLE table=default.Table_07709e08656c44c892d8a4f181d10bd0 [id=6253662e32044bea9e43124b223c3b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.591183 12958 tablet_service.cc:1467] Processing CreateTablet for tablet e32a40abc4e74586af79474e4b1675c5 (DEFAULT_TABLE table=default.Table_9362c95fb2df44f88a0a579b79b4ddd1 [id=c3a3b2e282314814a9bafaaa1cb33198]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.591706 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 0d94e3976a824bb9b10607d02764c8cc (DEFAULT_TABLE table=default.Table_a8f8c4d749444684a5b327c80c5f5829 [id=3e6deafd817f43c0b137bae39fb1d39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.592141 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e5305157e4a9cb6f38e403fed1333 (DEFAULT_TABLE table=default.Table_3dbe1bd9a46f4c9b8146d1be89e93c9d [id=40d6e63d54e44c089dff7b3115e15cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.594256 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 0d94e3976a824bb9b10607d02764c8cc (DEFAULT_TABLE table=default.Table_a8f8c4d749444684a5b327c80c5f5829 [id=3e6deafd817f43c0b137bae39fb1d39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.594458 13087 tablet_service.cc:1467] Processing CreateTablet for tablet e32a40abc4e74586af79474e4b1675c5 (DEFAULT_TABLE table=default.Table_9362c95fb2df44f88a0a579b79b4ddd1 [id=c3a3b2e282314814a9bafaaa1cb33198]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.594744 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d94e3976a824bb9b10607d02764c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.596865 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d9b3635582f347c8ba0cdc3fb54db89f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.597499 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9b3635582f347c8ba0cdc3fb54db89f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.597616 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e5305157e4a9cb6f38e403fed1333 (DEFAULT_TABLE table=default.Table_3dbe1bd9a46f4c9b8146d1be89e93c9d [id=40d6e63d54e44c089dff7b3115e15cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.598028 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e5305157e4a9cb6f38e403fed1333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.599078 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32a40abc4e74586af79474e4b1675c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.601642 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3cd77b3dbc674e2099537f5b46662b86" } new_table_name: "default.Table_dccf1c1b22014892a25a3ea6da1ecf29" modify_external_catalogs: true
I20240506 19:56:45.602488 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec5458abb234366af4c23754a54d68a (DEFAULT_TABLE table=default.Table_07709e08656c44c892d8a4f181d10bd0 [id=6253662e32044bea9e43124b223c3b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.602928 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec5458abb234366af4c23754a54d68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.604938 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 0d94e3976a824bb9b10607d02764c8cc (DEFAULT_TABLE table=default.Table_a8f8c4d749444684a5b327c80c5f5829 [id=3e6deafd817f43c0b137bae39fb1d39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.604975 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec5458abb234366af4c23754a54d68a (DEFAULT_TABLE table=default.Table_07709e08656c44c892d8a4f181d10bd0 [id=6253662e32044bea9e43124b223c3b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.605424 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91888e88234cbdbd00f48dc1574653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.605499 12812 tablet_service.cc:1467] Processing CreateTablet for tablet e32a40abc4e74586af79474e4b1675c5 (DEFAULT_TABLE table=default.Table_9362c95fb2df44f88a0a579b79b4ddd1 [id=c3a3b2e282314814a9bafaaa1cb33198]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.605870 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e5305157e4a9cb6f38e403fed1333 (DEFAULT_TABLE table=default.Table_3dbe1bd9a46f4c9b8146d1be89e93c9d [id=40d6e63d54e44c089dff7b3115e15cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.606664 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d94e3976a824bb9b10607d02764c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.606701 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_75a0ded1f5e044068dfb80dd28b1ae9b" } new_table_name: "default.Table_85577a3ca1fb430194567dcaa70fa260" modify_external_catalogs: true
I20240506 19:56:45.610770 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e5305157e4a9cb6f38e403fed1333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.612952 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_88a94bbd43f4495aac452adfdfa73c5d" } modify_external_catalogs: true
I20240506 19:56:45.613224 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_88a94bbd43f4495aac452adfdfa73c5d" } modify_external_catalogs: true
I20240506 19:56:45.614158 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32a40abc4e74586af79474e4b1675c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.615723 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5c4a9e274f4747b7aad740d422b75e23" } modify_external_catalogs: true
I20240506 19:56:45.615947 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5c4a9e274f4747b7aad740d422b75e23" } modify_external_catalogs: true
I20240506 19:56:45.617571 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9608d1af549c42e28d02df6bd141adf8" } modify_external_catalogs: true
I20240506 19:56:45.617722 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec5458abb234366af4c23754a54d68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.617813 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9608d1af549c42e28d02df6bd141adf8" } modify_external_catalogs: true
I20240506 19:56:45.623363 13243 ts_tablet_manager.cc:1888] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.623512 13243 log.cc:1196] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/4be929f0e72e4d22a9af2fe6b3072bbe
I20240506 19:56:45.623533 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec5458abb234366af4c23754a54d68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.623826 13243 ts_tablet_manager.cc:1909] T 4be929f0e72e4d22a9af2fe6b3072bbe P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:45.624106 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32a40abc4e74586af79474e4b1675c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.624528 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d94e3976a824bb9b10607d02764c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.624933 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e5305157e4a9cb6f38e403fed1333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.632969 13242 ts_tablet_manager.cc:1875] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.635181 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_58c604462fcb4b9cab62687d64877b02" } modify_external_catalogs: true
I20240506 19:56:45.635388 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_58c604462fcb4b9cab62687d64877b02" } modify_external_catalogs: true
I20240506 19:56:45.638460 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 4be929f0e72e4d22a9af2fe6b3072bbe (table default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]) successfully deleted
W20240506 19:56:45.638792 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet cd57563cc6c140c184d1725a2da6a475 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd57563cc6c140c184d1725a2da6a475 already in progress: creating tablet
W20240506 19:56:45.639048 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet da06187c83b94d03ae4ebb67a2873cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da06187c83b94d03ae4ebb67a2873cf0 already in progress: creating tablet
I20240506 19:56:45.639050 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.639200 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet db13b58d17c94e0f82052f823915d9af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:45.639293 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ea7e073296ce46e786362aaed96eb4dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7e073296ce46e786362aaed96eb4dd already in progress: creating tablet
I20240506 19:56:45.639418 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.640586 13243 tablet_replica.cc:330] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:45.640769 13243 raft_consensus.cc:2229] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.640905 13243 raft_consensus.cc:2258] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.641670 13243 ts_tablet_manager.cc:1875] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.646561 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 (DEFAULT_TABLE table=default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99 [id=c33f0589e1fc47bf8c672808d751ed5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.647024 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ea8e0a4ee47fea93eb4b36cdd2495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.652107 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9c19567619b64ee0a8dd637d7f69ce3e
I20240506 19:56:45.652664 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 9c19567619b64ee0a8dd637d7f69ce3e deleted and replaced by tablet a2e061a5a97646cc9b14ddf741c5b88e
I20240506 19:56:45.653270 12514 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 03b8cff0aa04487e95860b3c5ae407ba
I20240506 19:56:45.653846 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet dc206b31d3da458cb944d5f00e23d702
I20240506 19:56:45.653905 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 (DEFAULT_TABLE table=default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99 [id=c33f0589e1fc47bf8c672808d751ed5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.655768 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c19567619b64ee0a8dd637d7f69ce3e with delete_type TABLET_DATA_DELETED (replaced by tablet a2e061a5a97646cc9b14ddf741c5b88e) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.655937 12845 raft_consensus.cc:3037] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:45.656059 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet dc206b31d3da458cb944d5f00e23d702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.656432 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9944e5bfe0a741d69b8f9b6a9dad84dd
I20240506 19:56:45.656908 12519 catalog_manager.cc:6554] ReplaceTablet: tablet 9944e5bfe0a741d69b8f9b6a9dad84dd deleted and replaced by tablet a512993561e849239eca5c212f19af3f
I20240506 19:56:45.657884 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c19567619b64ee0a8dd637d7f69ce3e with delete_type TABLET_DATA_DELETED (replaced by tablet a2e061a5a97646cc9b14ddf741c5b88e) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.657927 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet dc206b31d3da458cb944d5f00e23d702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.658779 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd with delete_type TABLET_DATA_DELETED (replaced by tablet a512993561e849239eca5c212f19af3f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.660866 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd with delete_type TABLET_DATA_DELETED (replaced by tablet a512993561e849239eca5c212f19af3f) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:45.659241 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fce272e5b5f6483c9d031dff1104401b consensus state has no leader
I20240506 19:56:45.662762 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
I20240506 19:56:45.664754 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e54543d1544b1c8cff3e890e787337 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.664994 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet e17197adc6ca4593ac3fc494548a6e67 with delete_type TABLET_DATA_DELETED (replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.665230 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 1372be71e1d141ed89e58cdb806431a4 with delete_type TABLET_DATA_DELETED (replaced by tablet a1160ab972124630a9be7abeefb4aa56) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.665565 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ac9ecd2e034e49e49772db7435da3c81
W20240506 19:56:45.666121 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 449eeb67b602425bb40caa69b6a78a5a consensus state has no leader
I20240506 19:56:45.666568 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9ecd2e034e49e49772db7435da3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.667261 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9ecd2e034e49e49772db7435da3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.668735 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 (DEFAULT_TABLE table=default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99 [id=c33f0589e1fc47bf8c672808d751ed5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.669222 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ea8e0a4ee47fea93eb4b36cdd2495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.677151 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c19567619b64ee0a8dd637d7f69ce3e with delete_type TABLET_DATA_DELETED (replaced by tablet a2e061a5a97646cc9b14ddf741c5b88e) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.677613 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd with delete_type TABLET_DATA_DELETED (replaced by tablet a512993561e849239eca5c212f19af3f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.677875 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9ecd2e034e49e49772db7435da3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.678056 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet dc206b31d3da458cb944d5f00e23d702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.691519 13246 ts_tablet_manager.cc:1888] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.691802 13246 log.cc:1196] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b5a443c0d35741b9966b32a7808a97e2
I20240506 19:56:45.692394 13246 ts_tablet_manager.cc:1909] T b5a443c0d35741b9966b32a7808a97e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:45.694962 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c44481b32b644df187cba705ae9759c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.695701 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c44481b32b644df187cba705ae9759c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.697786 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 467c33f012584d489c2842e3e606f485 (DEFAULT_TABLE table=default.Table_b2476ed5d80041e68f12fa12ddef7008 [id=f410932ed8d9451bbdfe2550917a40af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.698395 13245 tablet_bootstrap.cc:492] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.698594 13245 ts_tablet_manager.cc:1367] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.218s user 0.004s sys 0.027s
I20240506 19:56:45.699429 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8c4e4e8dbeff40b6b23e5456dbb53729"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.699482 13245 raft_consensus.cc:348] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.699757 13245 raft_consensus.cc:374] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.699688 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f68578ef105f4a5b8170783670e18751" } new_table_name: "default.Table_03cf2611bf48455a925b28b842f1e351" modify_external_catalogs: true
I20240506 19:56:45.699834 13245 raft_consensus.cc:729] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:45.700152 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c4e4e8dbeff40b6b23e5456dbb53729 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.700078 13245 consensus_queue.cc:260] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.700665 13245 ts_tablet_manager.cc:1398] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:45.701046 13245 tablet_bootstrap.cc:492] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.701575 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f29a4d31b66342129de3cb8f5f56f136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.702116 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f29a4d31b66342129de3cb8f5f56f136 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.703765 13878 raft_consensus.cc:482] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.703989 13878 raft_consensus.cc:504] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.699450 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_94602d1502334dcb977668d7d42eebec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.704130 13245 tablet_bootstrap.cc:654] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.704841 13878 leader_election.cc:288] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:45.705351 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94602d1502334dcb977668d7d42eebec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:45.705896 13057 leader_election.cc:341] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:45.706773 13864 raft_consensus.cc:482] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.706921 13864 raft_consensus.cc:504] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.707530 13864 leader_election.cc:288] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.708738 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ec577dd06640068575288c24e1105a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:45.709342 12921 leader_election.cc:341] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.709906 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_46aaa890b4ce4d02919bab7b358c7a79" } new_table_name: "default.Table_ed6443bb846e47af83886d53a75f4e97" modify_external_catalogs: true
I20240506 19:56:45.710402 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9a458e9734bc4cd78dd0eb34dc07783f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.710930 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a458e9734bc4cd78dd0eb34dc07783f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.711274 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ec577dd06640068575288c24e1105a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:45.713912 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 581) took 1257 ms. Trace:
I20240506 19:56:45.714061 13246 rpcz_store.cc:276] 0506 19:56:44.456755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.457431 (+ 676us) service_pool.cc:225] Handling call
0506 19:56:45.517178 (+1059747us) ts_tablet_manager.cc:1120] Deleting tablet b5a443c0d35741b9966b32a7808a97e2
0506 19:56:45.517188 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.560278 (+ 43090us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.662850 (+102572us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.713859 (+ 51009us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028543}
I20240506 19:56:45.715384 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b5a443c0d35741b9966b32a7808a97e2 (table default.Table_1a5485067c1a4e529059f5434babbdc7 [id=54286945a7554349b48aa6511f84326c]) successfully deleted
I20240506 19:56:45.716650 13245 tablet_bootstrap.cc:492] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.716815 13245 ts_tablet_manager.cc:1367] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240506 19:56:45.717877 13245 raft_consensus.cc:348] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.718190 13245 raft_consensus.cc:374] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.718289 13245 raft_consensus.cc:729] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.718616 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_424aab7141bd4c80967b1104df604c32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.718650 13245 consensus_queue.cc:260] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:45.719269 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_424aab7141bd4c80967b1104df604c32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.719334 13245 ts_tablet_manager.cc:1398] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:45.719724 13245 tablet_bootstrap.cc:492] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.720649 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 580) took 1263 ms. Trace:
I20240506 19:56:45.721086 13246 rpcz_store.cc:276] 0506 19:56:44.456656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.457347 (+ 691us) service_pool.cc:225] Handling call
0506 19:56:45.719550 (+1262203us) ts_tablet_manager.cc:1120] Deleting tablet 1e22e5a015f74526b49363bc789cb4ed
0506 19:56:45.719560 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.719673 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1230117}
W20240506 19:56:45.723341 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1e22e5a015f74526b49363bc789cb4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e22e5a015f74526b49363bc789cb4ed already in progress: creating tablet
I20240506 19:56:45.723956 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 583) took 1266 ms. Trace:
I20240506 19:56:45.725610 13246 rpcz_store.cc:276] 0506 19:56:44.457017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.457489 (+ 472us) service_pool.cc:225] Handling call
0506 19:56:45.723828 (+1266339us) ts_tablet_manager.cc:1120] Deleting tablet 30ad527c3f2a4b53affa928e149db1df
0506 19:56:45.723839 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.723939 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226301}
I20240506 19:56:45.727252 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99" } modify_external_catalogs: true
I20240506 19:56:45.727402 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 467c33f012584d489c2842e3e606f485 (DEFAULT_TABLE table=default.Table_b2476ed5d80041e68f12fa12ddef7008 [id=f410932ed8d9451bbdfe2550917a40af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.727471 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ccc6d00ee45e4c4dbdbcf7403022cd99" } modify_external_catalogs: true
W20240506 19:56:45.729513 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 30ad527c3f2a4b53affa928e149db1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ad527c3f2a4b53affa928e149db1df already in progress: creating tablet
W20240506 19:56:45.730425 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7de7397db1e74b4488a3f9a70136bccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7de7397db1e74b4488a3f9a70136bccb already in progress: creating tablet
I20240506 19:56:45.721947 13245 tablet_bootstrap.cc:654] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.722882 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 467c33f012584d489c2842e3e606f485 (DEFAULT_TABLE table=default.Table_b2476ed5d80041e68f12fa12ddef7008 [id=f410932ed8d9451bbdfe2550917a40af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.731305 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467c33f012584d489c2842e3e606f485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.732026 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef799f2e474413d8f76f220a6bda665 (DEFAULT_TABLE table=default.Table_2fda57476820461185abfb4e54f9a3d9 [id=893f3cabb2d84a349345cc6fbf9f4656]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.732192 13246 tablet_replica.cc:330] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:45.732401 13246 raft_consensus.cc:2229] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.732573 13246 raft_consensus.cc:2258] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.737403 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467c33f012584d489c2842e3e606f485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.737649 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8335f9308046279a25a4a95d8e0f65" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:45.740854 13058 leader_election.cc:341] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.741009 13058 leader_election.cc:302] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.741485 13878 raft_consensus.cc:2726] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.745882 13245 tablet_bootstrap.cc:492] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.746038 13245 ts_tablet_manager.cc:1367] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240506 19:56:45.747490 13245 raft_consensus.cc:348] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.748533 13245 raft_consensus.cc:374] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.748589 13245 raft_consensus.cc:729] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.749416 13245 consensus_queue.cc:260] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.750689 13245 ts_tablet_manager.cc:1398] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:56:45.757236 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef799f2e474413d8f76f220a6bda665 (DEFAULT_TABLE table=default.Table_2fda57476820461185abfb4e54f9a3d9 [id=893f3cabb2d84a349345cc6fbf9f4656]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.757706 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef799f2e474413d8f76f220a6bda665 (DEFAULT_TABLE table=default.Table_2fda57476820461185abfb4e54f9a3d9 [id=893f3cabb2d84a349345cc6fbf9f4656]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.758122 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef799f2e474413d8f76f220a6bda665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.758312 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef799f2e474413d8f76f220a6bda665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.759719 13124 raft_consensus.cc:2454] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:45.760247 12922 leader_election.cc:302] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.761060 13864 raft_consensus.cc:2781] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:45.761144 13864 raft_consensus.cc:482] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:45.761201 13864 raft_consensus.cc:3037] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:45.765645 13245 tablet_bootstrap.cc:492] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.767791 13245 tablet_bootstrap.cc:654] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.767796 13944 raft_consensus.cc:482] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.768724 13944 raft_consensus.cc:504] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.769815 13944 leader_election.cc:288] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:45.771226 12781 leader_election.cc:341] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 5988b89cb52e4f858b52862138bed025
I20240506 19:56:45.774312 13246 ts_tablet_manager.cc:1875] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.776309 13911 consensus_queue.cc:1035] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:45.781728 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467c33f012584d489c2842e3e606f485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.782477 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef799f2e474413d8f76f220a6bda665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.782904 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ea8e0a4ee47fea93eb4b36cdd2495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.783077 13864 raft_consensus.cc:504] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.783815 13864 leader_election.cc:288] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.784521 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ec577dd06640068575288c24e1105a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:45.784744 13122 raft_consensus.cc:3037] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:45.787516 13122 raft_consensus.cc:2454] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:45.787999 12922 leader_election.cc:302] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:45.788249 13948 raft_consensus.cc:2781] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:45.788503 13948 raft_consensus.cc:686] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:45.788784 13948 consensus_queue.cc:237] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.789800 13864 raft_consensus.cc:482] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.789939 13864 raft_consensus.cc:504] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.790511 13864 leader_election.cc:288] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.793498 13864 raft_consensus.cc:482] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.793641 13864 raft_consensus.cc:504] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.794189 13864 leader_election.cc:288] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.795507 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4be565e2d4c59b3587baed72e4a9c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:45.795715 13123 raft_consensus.cc:2454] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:45.796203 12922 leader_election.cc:302] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:45.796531 13864 raft_consensus.cc:2781] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:45.796614 13864 raft_consensus.cc:482] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:45.796679 13864 raft_consensus.cc:3037] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:45.796617 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ec577dd06640068575288c24e1105a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:45.797390 12921 consensus_peers.cc:485] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:45.797529 12921 leader_election.cc:341] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.797924 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13b58d17c94e0f82052f823915d9af" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:45.798245 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4be565e2d4c59b3587baed72e4a9c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:45.801503 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13b58d17c94e0f82052f823915d9af" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:45.801739 13124 raft_consensus.cc:2454] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:45.802659 12922 leader_election.cc:302] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:45.804363 13911 raft_consensus.cc:2781] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:45.804448 13911 raft_consensus.cc:482] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:45.804522 13911 raft_consensus.cc:3037] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:45.805338 12921 leader_election.cc:341] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:45.805562 12921 leader_election.cc:341] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.807634 13887 raft_consensus.cc:482] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 059a5c4a3f7e49ef95af4b8341702745)
I20240506 19:56:45.807729 13955 tablet.cc:1722] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:45.807917 13887 raft_consensus.cc:504] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.810683 13887 leader_election.cc:288] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.811674 13085 tablet_service.cc:1467] Processing CreateTablet for tablet efc41afd84a74652a82cc639bf562d99 (DEFAULT_TABLE table=default.Table_3d6b94b6306c4200bfde515a4b2c418c [id=6b539c4e5b2444daa0f9ae05ed98ff54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.812126 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc41afd84a74652a82cc639bf562d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.813261 12963 tablet_service.cc:1467] Processing CreateTablet for tablet efc41afd84a74652a82cc639bf562d99 (DEFAULT_TABLE table=default.Table_3d6b94b6306c4200bfde515a4b2c418c [id=6b539c4e5b2444daa0f9ae05ed98ff54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.813573 12812 tablet_service.cc:1467] Processing CreateTablet for tablet efc41afd84a74652a82cc639bf562d99 (DEFAULT_TABLE table=default.Table_3d6b94b6306c4200bfde515a4b2c418c [id=6b539c4e5b2444daa0f9ae05ed98ff54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.813931 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc41afd84a74652a82cc639bf562d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.820988 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:45.774353 13245 tablet_bootstrap.cc:492] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.822712 13245 ts_tablet_manager.cc:1367] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240506 19:56:45.824443 13245 raft_consensus.cc:348] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.827178 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 418c79b0361f4b2480d4cfbadeef6dc8 (DEFAULT_TABLE table=default.Table_680d6530958740e293cd6eddcad88b21 [id=919e8b8b265b452e8972fa41f11645a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.828369 13245 raft_consensus.cc:374] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.829038 13245 raft_consensus.cc:729] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.829531 13245 consensus_queue.cc:260] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:45.831496 12783 leader_election.cc:341] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 5988b89cb52e4f858b52862138bed025
I20240506 19:56:45.827174 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 418c79b0361f4b2480d4cfbadeef6dc8 (DEFAULT_TABLE table=default.Table_680d6530958740e293cd6eddcad88b21 [id=919e8b8b265b452e8972fa41f11645a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.831113 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380bf30599d4130bbb80b4f26837779" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:45.831787 12783 leader_election.cc:302] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:45.832969 13944 raft_consensus.cc:2726] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.833194 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8c77e550ef40779b9c64d89df40a9a (DEFAULT_TABLE table=default.Table_456c37fe3dda48289f407577c6618ba9 [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.833969 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8c77e550ef40779b9c64d89df40a9a (DEFAULT_TABLE table=default.Table_456c37fe3dda48289f407577c6618ba9 [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.834328 13245 ts_tablet_manager.cc:1398] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240506 19:56:45.834355 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8c77e550ef40779b9c64d89df40a9a (DEFAULT_TABLE table=default.Table_456c37fe3dda48289f407577c6618ba9 [id=b164ee48c45e4826868242c03700533e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.834626 13245 tablet_bootstrap.cc:492] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.835055 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 418c79b0361f4b2480d4cfbadeef6dc8 (DEFAULT_TABLE table=default.Table_680d6530958740e293cd6eddcad88b21 [id=919e8b8b265b452e8972fa41f11645a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:45.836781 12783 leader_election.cc:302] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:45.836871 13245 tablet_bootstrap.cc:654] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.837183 13944 raft_consensus.cc:2726] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:45.863010 13245 tablet_bootstrap.cc:492] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.863207 13245 ts_tablet_manager.cc:1367] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.001s
I20240506 19:56:45.870165 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8c77e550ef40779b9c64d89df40a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.870872 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc41afd84a74652a82cc639bf562d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.871384 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418c79b0361f4b2480d4cfbadeef6dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.873407 13242 ts_tablet_manager.cc:1888] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.873396 13245 raft_consensus.cc:348] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.873581 13242 log.cc:1196] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2d9230ed2e9a49ecb0a5e5d520cd03aa
I20240506 19:56:45.873637 13245 raft_consensus.cc:374] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.873704 13245 raft_consensus.cc:729] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.873994 13242 ts_tablet_manager.cc:1909] T 2d9230ed2e9a49ecb0a5e5d520cd03aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:45.874354 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418c79b0361f4b2480d4cfbadeef6dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.874444 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8c77e550ef40779b9c64d89df40a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.870122 13243 ts_tablet_manager.cc:1888] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.874941 13243 log.cc:1196] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:45.875030 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418c79b0361f4b2480d4cfbadeef6dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.875140 13245 consensus_queue.cc:260] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.875277 13243 ts_tablet_manager.cc:1909] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:45.875689 13245 ts_tablet_manager.cc:1398] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240506 19:56:45.876035 13245 tablet_bootstrap.cc:492] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.876087 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8c77e550ef40779b9c64d89df40a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:45.886031 13245 tablet_bootstrap.cc:654] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:45.888049 12507 consensus_peers.cc:485] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d -> Peer 97f390a786034cddbff757779037f861 (127.12.48.252:36867): Couldn't send request to peer 97f390a786034cddbff757779037f861. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240506 19:56:45.889374 13864 raft_consensus.cc:504] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.890105 13864 leader_election.cc:288] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.890969 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4be565e2d4c59b3587baed72e4a9c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:45.891155 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4be565e2d4c59b3587baed72e4a9c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:45.891441 13122 raft_consensus.cc:3037] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:45.891615 12921 leader_election.cc:341] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.901938 13245 tablet_bootstrap.cc:492] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:45.902179 13245 ts_tablet_manager.cc:1367] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240506 19:56:45.903281 13245 raft_consensus.cc:348] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.903662 13245 raft_consensus.cc:374] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:45.903769 13245 raft_consensus.cc:729] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:45.904047 13245 consensus_queue.cc:260] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.904583 13245 ts_tablet_manager.cc:1398] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:45.904901 13245 tablet_bootstrap.cc:492] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:45.906797 13245 tablet_bootstrap.cc:654] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:45.914386 13243 tablet_replica.cc:330] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:45.915112 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 (table default.Table_bece2f866f8449ad8ef2f48007dc14ef [id=4021900d99ce4302b2de9b33a687e94f]) successfully deleted
I20240506 19:56:45.922552 13878 raft_consensus.cc:482] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:45.922823 13878 raft_consensus.cc:504] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.923599 13878 leader_election.cc:288] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:45.925334 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c56ef126a4ca48b6a379a6771bc1c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:45.925937 13058 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:45.927901 13057 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:45.928025 13057 leader_election.cc:302] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.928655 13941 raft_consensus.cc:2726] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:45.938759 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 796beb8e0a2c41d698bd4c05796263f1
I20240506 19:56:45.939965 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.940229 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
I20240506 19:56:45.940402 12515 catalog_manager.cc:5484] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:45.940979 12515 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:45.941840 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.941740 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.942337 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.942255 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:45.942937 13246 ts_tablet_manager.cc:1888] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:45.943065 13246 log.cc:1196] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/00de09ddcdd948a6a3500a6c9d814c14
I20240506 19:56:45.943326 13955 tablet.cc:1722] T e1f1e4efbe8b40b199fcb3020569e1a1 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:45.943603 13246 ts_tablet_manager.cc:1909] T 00de09ddcdd948a6a3500a6c9d814c14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:45.943989 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 450) took 3526 ms. Trace:
I20240506 19:56:45.944092 13242 rpcz_store.cc:276] 0506 19:56:42.417131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.417366 (+ 235us) service_pool.cc:225] Handling call
0506 19:56:45.567137 (+3149771us) ts_tablet_manager.cc:1120] Deleting tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa
0506 19:56:45.567147 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.807123 (+239976us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.873236 (+ 66113us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.943931 (+ 70695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3149522}
I20240506 19:56:45.944631 13242 tablet_replica.cc:330] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:45.944875 13242 raft_consensus.cc:2229] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:45.945376 13242 raft_consensus.cc:2258] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.946218 13242 ts_tablet_manager.cc:1875] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.946838 13911 raft_consensus.cc:504] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.947587 13911 leader_election.cc:288] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:45.943969 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2486dda60f0b4f2b9a63dbf17885bc5e" } new_table_name: "default.Table_66fa4fdbd72742d9aa51e8911f3b72cc" modify_external_catalogs: true
I20240506 19:56:45.949396 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13b58d17c94e0f82052f823915d9af" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:45.946027 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1f6bae23516f4e79b0662b14c8a8ea3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:45.950481 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f6bae23516f4e79b0662b14c8a8ea3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:45.952029 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:45.953648 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2d9230ed2e9a49ecb0a5e5d520cd03aa (table default.Table_bcd38427a79c48d1badd4f2536ff238f [id=f23583e607a4419986d06a38c45bbdf5]) successfully deleted
I20240506 19:56:45.954154 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 159ea8e0a4ee47fea93eb4b36cdd2495
W20240506 19:56:45.955004 12921 leader_election.cc:341] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:45.954973 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_11b0e013ac1943e3ad3061b8a451c8d8" } modify_external_catalogs: true
I20240506 19:56:45.955148 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_11b0e013ac1943e3ad3061b8a451c8d8" } modify_external_catalogs: true
I20240506 19:56:45.955207 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:45.957249 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_aded0cd817a0451cbbb77322ae0b30b0" } modify_external_catalogs: true
I20240506 19:56:45.957463 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_aded0cd817a0451cbbb77322ae0b30b0" } modify_external_catalogs: true
I20240506 19:56:45.958631 13243 raft_consensus.cc:2229] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.959071 13243 raft_consensus.cc:2258] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.959677 13243 ts_tablet_manager.cc:1875] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:45.961686 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 902d43d15d334939bddb254ae04ca174 consensus state has no leader
I20240506 19:56:45.962185 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_840b5efc8ccf4fef961df12c95e98177" } new_table_name: "default.Table_12562fde5f794337b741599bd5fc9ba0" modify_external_catalogs: true
I20240506 19:56:45.962601 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.962821 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:45.966210 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6f7629baef0946e6aa92b1cbdef4d362" } modify_external_catalogs: true
I20240506 19:56:45.966415 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6f7629baef0946e6aa92b1cbdef4d362" } modify_external_catalogs: true
W20240506 19:56:45.968323 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a568db06324a4fd1b58a29ef0831eb71 consensus state has no leader
I20240506 19:56:45.968705 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bd20aca236004f33954f66b72b4864e6" } new_table_name: "default.Table_5c08c51b4a7848aa8b49a78e020d7d6c" modify_external_catalogs: true
I20240506 19:56:45.970580 13122 raft_consensus.cc:2454] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:45.971725 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 590) took 1139 ms. Trace:
I20240506 19:56:45.971840 13246 rpcz_store.cc:276] 0506 19:56:44.832532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.832611 (+ 79us) service_pool.cc:225] Handling call
0506 19:56:45.732135 (+899524us) ts_tablet_manager.cc:1120] Deleting tablet 00de09ddcdd948a6a3500a6c9d814c14
0506 19:56:45.732145 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:45.870410 (+138265us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.942797 (+ 72387us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:45.971701 (+ 28904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":893922}
I20240506 19:56:45.972173 13246 tablet_replica.cc:330] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:45.972335 13246 raft_consensus.cc:2229] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:45.972474 13246 raft_consensus.cc:2258] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:45.972740 12922 leader_election.cc:302] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:45.973017 13246 ts_tablet_manager.cc:1875] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:45.974081 13911 raft_consensus.cc:2781] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:45.974226 13911 raft_consensus.cc:686] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:45.974469 13911 consensus_queue.cc:237] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:45.976312 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:45.977280 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 00de09ddcdd948a6a3500a6c9d814c14 (table default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]) successfully deleted
I20240506 19:56:45.984194 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db13b58d17c94e0f82052f823915d9af" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:45.984431 13124 raft_consensus.cc:3037] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:45.994719 13965 tablet.cc:1722] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:45.994700 13245 tablet_bootstrap.cc:492] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.000757 13245 ts_tablet_manager.cc:1367] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.008s
I20240506 19:56:46.001798 13245 raft_consensus.cc:348] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.002125 13245 raft_consensus.cc:374] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.002317 13245 raft_consensus.cc:729] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.002720 13245 consensus_queue.cc:260] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.003403 13245 ts_tablet_manager.cc:1398] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:46.003813 13245 tablet_bootstrap.cc:492] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.007792 13245 tablet_bootstrap.cc:654] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.020007 13878 raft_consensus.cc:482] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.020211 13878 raft_consensus.cc:504] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.020972 13878 leader_election.cc:288] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:46.022329 13057 leader_election.cc:341] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 4be929f0e72e4d22a9af2fe6b3072bbe
I20240506 19:56:46.022567 13941 raft_consensus.cc:482] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.022698 13941 raft_consensus.cc:504] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.023993 13941 leader_election.cc:288] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:46.024495 13057 leader_election.cc:341] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:46.031458 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be929f0e72e4d22a9af2fe6b3072bbe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.031998 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c1590e36b4471d8e72f4b07410ee0f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.032703 13058 leader_election.cc:341] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.032827 13058 leader_election.cc:302] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:46.033231 13058 leader_election.cc:341] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.033341 13058 leader_election.cc:302] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.033960 13878 raft_consensus.cc:2726] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.034176 13878 raft_consensus.cc:2726] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.039932 13966 tablet.cc:1722] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:46.046967 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c0c7cae21c714148a37cab389d116e91
W20240506 19:56:46.047505 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd6245e4adce4038ad6f988877d75045 consensus state has no leader
I20240506 19:56:46.048707 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c7cae21c714148a37cab389d116e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.048717 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fb3b18d9d707453393cfb058b284bfa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.049389 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb3b18d9d707453393cfb058b284bfa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.051113 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c7cae21c714148a37cab389d116e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.058063 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c7cae21c714148a37cab389d116e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.062978 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
W20240506 19:56:46.063468 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d43da1d12ce647dcbcd31263932b2f54 consensus state has no leader
I20240506 19:56:46.063759 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4be4e628b9364cccb525b8e1b8070286
I20240506 19:56:46.064980 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:46.064967 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be4e628b9364cccb525b8e1b8070286 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.065039 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be4e628b9364cccb525b8e1b8070286 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.065352 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]). Expected version 3 got 0
I20240506 19:56:46.065928 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be4e628b9364cccb525b8e1b8070286 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.067200 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet e17197adc6ca4593ac3fc494548a6e67 with delete_type TABLET_DATA_DELETED (replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.067788 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.068305 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.070559 13245 tablet_bootstrap.cc:492] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.070749 13245 ts_tablet_manager.cc:1367] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.007s
I20240506 19:56:46.071445 13245 raft_consensus.cc:348] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.071640 13245 raft_consensus.cc:374] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.071697 13245 raft_consensus.cc:729] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.072206 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 1372be71e1d141ed89e58cdb806431a4 with delete_type TABLET_DATA_DELETED (replaced by tablet a1160ab972124630a9be7abeefb4aa56) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.073005 13124 raft_consensus.cc:2454] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:46.074167 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7bc30f4690b24c5ea8ab2f7a26122992
W20240506 19:56:46.074975 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2be78cbc8264f6e8d2e05137135161c consensus state has no leader
I20240506 19:56:46.075810 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc30f4690b24c5ea8ab2f7a26122992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.076418 12922 leader_election.cc:302] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:46.076656 12922 raft_consensus.cc:2670] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:46.077574 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc30f4690b24c5ea8ab2f7a26122992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.078570 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc30f4690b24c5ea8ab2f7a26122992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.079071 13944 raft_consensus.cc:482] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.079265 13944 raft_consensus.cc:504] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.080016 13944 leader_election.cc:288] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.081036 12781 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 27b24fa4663244a88dac9ea99fe38f3a
W20240506 19:56:46.087196 13249 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.014s sys 0.053s
W20240506 19:56:46.087314 13249 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.512s user 0.014s sys 0.053s
W20240506 19:56:46.088107 13244 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.053s sys 0.088s
W20240506 19:56:46.088214 13244 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.509s user 0.053s sys 0.088s
I20240506 19:56:46.089026 12519 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet e1f1e4efbe8b40b199fcb3020569e1a1 (table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]). Expected version 3 got 0
I20240506 19:56:46.090795 13249 tablet_bootstrap.cc:492] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.090938 13249 ts_tablet_manager.cc:1367] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.519s user 0.014s sys 0.055s
I20240506 19:56:46.091773 13249 raft_consensus.cc:348] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.092020 13249 raft_consensus.cc:374] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.092092 13249 raft_consensus.cc:729] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.092511 13911 raft_consensus.cc:482] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.092619 13911 raft_consensus.cc:504] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.093839 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1372be71e1d141ed89e58cdb806431a4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.094460 12921 leader_election.cc:341] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.094580 13244 tablet_bootstrap.cc:492] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:46.094755 13244 ts_tablet_manager.cc:1367] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.518s user 0.055s sys 0.089s
I20240506 19:56:46.095391 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1372be71e1d141ed89e58cdb806431a4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.095609 13122 raft_consensus.cc:2454] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:46.096055 12922 leader_election.cc:302] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:46.096155 12783 leader_election.cc:341] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 27b24fa4663244a88dac9ea99fe38f3a
I20240506 19:56:46.096282 12783 leader_election.cc:302] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.096323 13864 raft_consensus.cc:2781] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.096385 13864 raft_consensus.cc:482] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.096443 13864 raft_consensus.cc:3037] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.096783 13944 raft_consensus.cc:2726] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.098364 13911 leader_election.cc:288] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.092370 13249 consensus_queue.cc:260] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.099125 13249 ts_tablet_manager.cc:1398] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:56:46.101351 13244 raft_consensus.cc:348] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.103708 13249 tablet_bootstrap.cc:492] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.104425 13244 raft_consensus.cc:374] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.104640 13244 raft_consensus.cc:729] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.105119 13244 consensus_queue.cc:260] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:46.113550 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 796628a0859c4511a8cb35e495dc8886 consensus state has no leader
I20240506 19:56:46.119434 13249 tablet_bootstrap.cc:654] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.128422 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 796beb8e0a2c41d698bd4c05796263f1 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.133378 13244 ts_tablet_manager.cc:1398] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.039s user 0.002s sys 0.001s
I20240506 19:56:46.134843 13245 consensus_queue.cc:260] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.133790 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet b529bb9362924caca399491e7d9e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.141968 13244 tablet_bootstrap.cc:492] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:46.142441 13245 ts_tablet_manager.cc:1398] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.072s user 0.001s sys 0.001s
I20240506 19:56:46.146816 13864 raft_consensus.cc:504] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.147446 13864 leader_election.cc:288] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.147859 13243 ts_tablet_manager.cc:1888] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.147964 13243 log.cc:1196] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/db13b58d17c94e0f82052f823915d9af
I20240506 19:56:46.148286 13243 ts_tablet_manager.cc:1909] T db13b58d17c94e0f82052f823915d9af P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:46.143114 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e54543d1544b1c8cff3e890e787337 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.145351 13246 ts_tablet_manager.cc:1888] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.155808 13246 log.cc:1196] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/0e2005ef58fd4fcea16abe771ee385f8
I20240506 19:56:46.156136 13246 ts_tablet_manager.cc:1909] T 0e2005ef58fd4fcea16abe771ee385f8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:46.152977 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1372be71e1d141ed89e58cdb806431a4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:46.156488 13124 raft_consensus.cc:3037] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.160916 13245 tablet_bootstrap.cc:492] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.163686 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e71ba8bf0b874298b72cae27e8607c6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.164315 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e71ba8bf0b874298b72cae27e8607c6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.164999 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cda4f793e543441f94c07714c44ad670" } modify_external_catalogs: true
I20240506 19:56:46.165180 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cda4f793e543441f94c07714c44ad670" } modify_external_catalogs: true
I20240506 19:56:46.165560 13242 ts_tablet_manager.cc:1888] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240506 19:56:46.166011 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410" } modify_external_catalogs: true
I20240506 19:56:46.166378 13864 raft_consensus.cc:482] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.166631 13242 log.cc:1196] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d5725b47e43244f0af881b07e145b3aa
I20240506 19:56:46.166528 13864 raft_consensus.cc:504] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.167269 13864 leader_election.cc:288] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.168116 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17197adc6ca4593ac3fc494548a6e67" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.168342 13122 raft_consensus.cc:2454] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:46.168908 13941 raft_consensus.cc:482] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.168900 12922 leader_election.cc:302] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:46.169027 13941 raft_consensus.cc:504] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.169211 13864 raft_consensus.cc:2781] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.169291 13864 raft_consensus.cc:482] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.169377 13864 raft_consensus.cc:3037] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.169616 13941 leader_election.cc:288] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.172214 13249 tablet_bootstrap.cc:492] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.172385 13249 ts_tablet_manager.cc:1367] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.003s
I20240506 19:56:46.173146 13249 raft_consensus.cc:348] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.173384 13249 raft_consensus.cc:374] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.173451 13249 raft_consensus.cc:729] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.173683 13249 consensus_queue.cc:260] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:46.175138 13057 leader_election.cc:341] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:46.176362 13249 ts_tablet_manager.cc:1398] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:46.177098 13249 tablet_bootstrap.cc:492] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.187983 13249 tablet_bootstrap.cc:654] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.168445 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410" } modify_external_catalogs: true
I20240506 19:56:46.191016 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b8b3b6c37e2d4fe9a43050bea37c80b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.168557 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2c27273f191c449d882ac1e69af5a4e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.191982 13243 tablet_replica.cc:330] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:46.192175 13243 raft_consensus.cc:2229] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.192595 13243 raft_consensus.cc:2258] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.193219 13243 ts_tablet_manager.cc:1875] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.194147 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ed6443bb846e47af83886d53a75f4e97" } new_table_name: "default.Table_767d7448a8524f9c9a96fff950481033" modify_external_catalogs: true
I20240506 19:56:46.166440 13245 tablet_bootstrap.cc:654] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.195516 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_971fe431636342168d3237de7c0a271d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.196166 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_971fe431636342168d3237de7c0a271d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.197661 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet db13b58d17c94e0f82052f823915d9af (table default.Table_bb8517f5ef13428c9dcfc8c72d0cab0e [id=21f83b4858344bf9959aee15e02a7bf7]) successfully deleted
W20240506 19:56:46.198629 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8b3b6c37e2d4fe9a43050bea37c80b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:46.199016 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 00de09ddcdd948a6a3500a6c9d814c14 because the tablet was not found. No further retry: Not found: Tablet not found: 00de09ddcdd948a6a3500a6c9d814c14
W20240506 19:56:46.199294 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9c19567619b64ee0a8dd637d7f69ce3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c19567619b64ee0a8dd637d7f69ce3e already in progress: creating tablet
W20240506 19:56:46.201438 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet dc206b31d3da458cb944d5f00e23d702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc206b31d3da458cb944d5f00e23d702 already in progress: creating tablet
W20240506 19:56:46.201746 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9944e5bfe0a741d69b8f9b6a9dad84dd already in progress: creating tablet
I20240506 19:56:46.201951 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0027f2a42b1a494685c82f0862604186"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.203190 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0027f2a42b1a494685c82f0862604186 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.204299 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_917d34debc2543bdb5c53f1cad733526"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.207195 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17197adc6ca4593ac3fc494548a6e67" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.207737 13242 ts_tablet_manager.cc:1909] T d5725b47e43244f0af881b07e145b3aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:46.207927 12921 leader_election.cc:341] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.208127 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 451) took 3790 ms. Trace:
I20240506 19:56:46.200143 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1372be71e1d141ed89e58cdb806431a4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:46.208698 12921 leader_election.cc:341] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.208683 13242 rpcz_store.cc:276] 0506 19:56:42.417312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.418075 (+ 763us) service_pool.cc:225] Handling call
0506 19:56:45.944533 (+3526458us) ts_tablet_manager.cc:1120] Deleting tablet d5725b47e43244f0af881b07e145b3aa
0506 19:56:45.944545 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.043609 (+ 99064us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.146542 (+102933us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.208083 (+ 61541us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3526258}
I20240506 19:56:46.209492 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d04a1c5ee7b4d529fa2f4da4945ff09" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.217025 13245 tablet_bootstrap.cc:492] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.217327 13245 ts_tablet_manager.cc:1367] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.000s
I20240506 19:56:46.218197 13245 raft_consensus.cc:348] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.218488 13245 raft_consensus.cc:374] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.218588 13245 raft_consensus.cc:729] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:46.218662 13058 leader_election.cc:341] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.218793 13058 leader_election.cc:302] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.218863 13245 consensus_queue.cc:260] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.219172 13941 raft_consensus.cc:2726] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.219408 13245 ts_tablet_manager.cc:1398] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:46.219678 13245 tablet_bootstrap.cc:492] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.221278 13245 tablet_bootstrap.cc:654] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:46.227097 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_917d34debc2543bdb5c53f1cad733526 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.229235 13249 tablet_bootstrap.cc:492] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.229423 13249 ts_tablet_manager.cc:1367] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240506 19:56:46.230829 13249 raft_consensus.cc:348] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.231101 13249 raft_consensus.cc:374] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.231158 13249 raft_consensus.cc:729] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:46.231472 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c27273f191c449d882ac1e69af5a4e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.232331 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d5725b47e43244f0af881b07e145b3aa (table default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]) successfully deleted
I20240506 19:56:46.233999 13878 raft_consensus.cc:482] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.234143 13878 raft_consensus.cc:504] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.234820 13878 leader_election.cc:288] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.235589 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e54543d1544b1c8cff3e890e787337" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:46.235749 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 458) took 3646 ms. Trace:
I20240506 19:56:46.236160 13242 rpcz_store.cc:276] 0506 19:56:42.562425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.562475 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:46.208968 (+3646493us) ts_tablet_manager.cc:1120] Deleting tablet 4dcc036a20824736b16815bcb47645c4
0506 19:56:46.208978 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.209068 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3645855}
I20240506 19:56:46.236639 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 461) took 3653 ms. Trace:
W20240506 19:56:46.236624 13057 leader_election.cc:341] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: Tablet not RUNNING: STOPPED
W20240506 19:56:46.237133 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4dcc036a20824736b16815bcb47645c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dcc036a20824736b16815bcb47645c4 already in progress: creating tablet
I20240506 19:56:46.237557 13242 rpcz_store.cc:276] 0506 19:56:42.583139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.583209 (+ 70us) service_pool.cc:225] Handling call
0506 19:56:46.236468 (+3653259us) ts_tablet_manager.cc:1120] Deleting tablet bff7050bbc2e47cf823d25ec10b89718
0506 19:56:46.236481 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.236593 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3640870}
W20240506 19:56:46.238175 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bff7050bbc2e47cf823d25ec10b89718 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bff7050bbc2e47cf823d25ec10b89718 already in progress: creating tablet
I20240506 19:56:46.238481 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 465) took 3622 ms. Trace:
I20240506 19:56:46.238646 13242 rpcz_store.cc:276] 0506 19:56:42.615897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.616532 (+ 635us) service_pool.cc:225] Handling call
0506 19:56:46.237832 (+3621300us) ts_tablet_manager.cc:1120] Deleting tablet c8ece0bd3a3b47ef88a0b4910f0efae9
0506 19:56:46.237843 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.238448 (+ 605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3621060}
I20240506 19:56:46.239032 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 466) took 3618 ms. Trace:
W20240506 19:56:46.239493 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c8ece0bd3a3b47ef88a0b4910f0efae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8ece0bd3a3b47ef88a0b4910f0efae9 already in progress: creating tablet
I20240506 19:56:46.239849 13242 rpcz_store.cc:276] 0506 19:56:42.620142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.620206 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:46.238907 (+3618701us) ts_tablet_manager.cc:1120] Deleting tablet db13b58d17c94e0f82052f823915d9af
0506 19:56:46.238915 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.239002 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3618493}
W20240506 19:56:46.240829 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet db13b58d17c94e0f82052f823915d9af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db13b58d17c94e0f82052f823915d9af already in progress: creating tablet
I20240506 19:56:46.240236 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 467) took 3574 ms. Trace:
I20240506 19:56:46.240314 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e54543d1544b1c8cff3e890e787337" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.242769 13058 leader_election.cc:341] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.242899 13058 leader_election.cc:302] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.242847 13242 rpcz_store.cc:276] 0506 19:56:42.666271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.666860 (+ 589us) service_pool.cc:225] Handling call
0506 19:56:46.240115 (+3573255us) ts_tablet_manager.cc:1120] Deleting tablet ecc88443c04b42d19cc7fba339eeeaf5
0506 19:56:46.240123 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.240204 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3559814}
I20240506 19:56:46.242898 13249 consensus_queue.cc:260] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.243176 13242 tablet_replica.cc:330] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:46.244763 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ed6b424598be490999d5da821583c6b6" } new_table_name: "default.Table_fbd2d442ecdb4935bdff3a94b68cc073" modify_external_catalogs: true
I20240506 19:56:46.247633 13124 raft_consensus.cc:2454] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:46.248054 13864 raft_consensus.cc:504] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.248070 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6d330fca725b4de09fa2e8a21d67d676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.248760 13864 leader_election.cc:288] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.249102 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet cd6245e4adce4038ad6f988877d75045 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.252341 13941 raft_consensus.cc:2726] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.252535 13941 raft_consensus.cc:482] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.252646 13941 raft_consensus.cc:504] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.252884 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5" } modify_external_catalogs: true
I20240506 19:56:46.253418 13941 leader_election.cc:288] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.253692 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5" } modify_external_catalogs: true
I20240506 19:56:46.254168 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 589) took 1421 ms. Trace:
I20240506 19:56:46.254258 13246 rpcz_store.cc:276] 0506 19:56:44.832383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.832605 (+ 222us) service_pool.cc:225] Handling call
0506 19:56:45.972125 (+1139520us) ts_tablet_manager.cc:1120] Deleting tablet 0e2005ef58fd4fcea16abe771ee385f8
0506 19:56:45.972135 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.049521 (+ 77386us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.145187 (+ 95666us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.254144 (+108957us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1379,"tablet-delete.queue_time_us":1133766}
I20240506 19:56:46.254855 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 598) took 1344 ms. Trace:
I20240506 19:56:46.254948 13246 rpcz_store.cc:276] 0506 19:56:44.909979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.910162 (+ 183us) service_pool.cc:225] Handling call
0506 19:56:46.254618 (+1344456us) ts_tablet_manager.cc:1120] Deleting tablet 03df4632adfa4b0ab999aeaa5bb4e1f4
0506 19:56:46.254631 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.254837 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344203}
I20240506 19:56:46.255316 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 597) took 1345 ms. Trace:
I20240506 19:56:46.255396 13246 rpcz_store.cc:276] 0506 19:56:44.909866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.910827 (+ 961us) service_pool.cc:225] Handling call
0506 19:56:46.255193 (+1344366us) ts_tablet_manager.cc:1120] Deleting tablet 04738def56324a368f8182543107e32e
0506 19:56:46.255203 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.255302 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344126}
I20240506 19:56:46.255522 13242 raft_consensus.cc:2229] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.255728 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 600) took 1322 ms. Trace:
I20240506 19:56:46.255738 13242 raft_consensus.cc:2258] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.255796 13246 rpcz_store.cc:276] 0506 19:56:44.933132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.933187 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:46.255614 (+1322427us) ts_tablet_manager.cc:1120] Deleting tablet 09cb71b5962f4561bf0fdbe12bf45296
0506 19:56:46.255623 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.255713 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321715}
I20240506 19:56:46.256189 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 601) took 1322 ms. Trace:
I20240506 19:56:46.256263 13246 rpcz_store.cc:276] 0506 19:56:44.933224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.933274 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:46.256042 (+1322768us) ts_tablet_manager.cc:1120] Deleting tablet ca23d6f0a9e0445cbd4d0e708c8de2d3
0506 19:56:46.256051 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.256175 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322004}
I20240506 19:56:46.256351 13242 ts_tablet_manager.cc:1875] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.256587 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 599) took 1323 ms. Trace:
I20240506 19:56:46.256660 13246 rpcz_store.cc:276] 0506 19:56:44.932969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.937394 (+ 4425us) service_pool.cc:225] Handling call
0506 19:56:46.256481 (+1319087us) ts_tablet_manager.cc:1120] Deleting tablet 50ef5a2799b447f992207452dbe39645
0506 19:56:46.256490 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.256572 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318893}
I20240506 19:56:46.256981 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 602) took 1304 ms. Trace:
I20240506 19:56:46.257050 13246 rpcz_store.cc:276] 0506 19:56:44.952042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.952096 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:46.256884 (+1304788us) ts_tablet_manager.cc:1120] Deleting tablet 9df3ab9dff5148f9bb335a7209b29bbc
0506 19:56:46.256892 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.256967 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304521}
I20240506 19:56:46.257231 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17197adc6ca4593ac3fc494548a6e67" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:46.257807 12921 leader_election.cc:341] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.258034 13244 tablet_bootstrap.cc:654] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.258280 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 0e2005ef58fd4fcea16abe771ee385f8 (table default.Table_577b201c2bf345bdbaeb7bda56b8296c [id=0772215de50d467585bec1f234187f97]) successfully deleted
W20240506 19:56:46.258600 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 already in progress: creating tablet
W20240506 19:56:46.258833 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 04738def56324a368f8182543107e32e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04738def56324a368f8182543107e32e already in progress: creating tablet
W20240506 19:56:46.259042 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 09cb71b5962f4561bf0fdbe12bf45296 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09cb71b5962f4561bf0fdbe12bf45296 already in progress: creating tablet
W20240506 19:56:46.259217 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 already in progress: creating tablet
I20240506 19:56:46.259222 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.259421 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 50ef5a2799b447f992207452dbe39645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ef5a2799b447f992207452dbe39645 already in progress: creating tablet
W20240506 19:56:46.259635 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9df3ab9dff5148f9bb335a7209b29bbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9df3ab9dff5148f9bb335a7209b29bbc already in progress: creating tablet
W20240506 19:56:46.260031 13058 leader_election.cc:341] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.260144 13246 tablet_replica.cc:330] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:46.261382 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d9b3635582f347c8ba0cdc3fb54db89f" } new_table_name: "default.Table_31d45289fb564acaa4d5f2e45801473d" modify_external_catalogs: true
I20240506 19:56:46.261797 12922 leader_election.cc:302] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.262270 13864 raft_consensus.cc:2781] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:46.262399 13864 raft_consensus.cc:686] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:46.262610 13246 raft_consensus.cc:2229] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.262794 13246 raft_consensus.cc:2258] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.262651 13864 consensus_queue.cc:237] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.263603 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.264196 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:46.264364 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d330fca725b4de09fa2e8a21d67d676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.264369 12986 raft_consensus.cc:2454] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:46.264801 13057 leader_election.cc:302] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.265215 13878 raft_consensus.cc:2781] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.265287 13878 raft_consensus.cc:482] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.265370 13878 raft_consensus.cc:3037] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:46.243903 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ecc88443c04b42d19cc7fba339eeeaf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecc88443c04b42d19cc7fba339eeeaf5 already in progress: creating tablet
I20240506 19:56:46.263309 13246 ts_tablet_manager.cc:1875] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.270104 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17197adc6ca4593ac3fc494548a6e67" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:46.270362 13124 raft_consensus.cc:3037] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:46.273623 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a568db06324a4fd1b58a29ef0831eb71 consensus state has no leader
I20240506 19:56:46.274128 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_92e6b60423294d189484e6f67d5eb1ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.274760 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92e6b60423294d189484e6f67d5eb1ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.268469 13249 ts_tablet_manager.cc:1398] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240506 19:56:46.279502 13249 tablet_bootstrap.cc:492] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:46.281780 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 59930.6us
W20240506 19:56:46.285251 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46916e778bf34188959e1cfff084bffc consensus state has no leader
I20240506 19:56:46.285887 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_092635dc2dec452490c7710c0cc1a37e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.286522 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_092635dc2dec452490c7710c0cc1a37e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.287899 13944 raft_consensus.cc:482] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.288149 13944 raft_consensus.cc:504] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.289098 13944 leader_election.cc:288] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.290278 12783 leader_election.cc:341] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d6c656e16c1045dea79bc23b64fdd5e8
W20240506 19:56:46.290508 12781 leader_election.cc:341] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d6c656e16c1045dea79bc23b64fdd5e8
I20240506 19:56:46.290726 12781 leader_election.cc:302] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.291115 13944 raft_consensus.cc:2726] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.302017 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_53cda80d706841ef876c3a0a4b51271a" } new_table_name: "default.Table_e13e6c9a0cd1442db0f3cb1b4c6cb5f4" modify_external_catalogs: true
W20240506 19:56:46.302925 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7deb29db8a2340a786edc11e3e370077 has no consensus state
I20240506 19:56:46.303469 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dd9774c013d441f4b9fc4a0afdf7daa2" } modify_external_catalogs: true
I20240506 19:56:46.303740 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dd9774c013d441f4b9fc4a0afdf7daa2" } modify_external_catalogs: true
I20240506 19:56:46.302017 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8937af041a2e4c8faa1bc31f4d1e4b05" } modify_external_catalogs: true
I20240506 19:56:46.305054 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8937af041a2e4c8faa1bc31f4d1e4b05" } modify_external_catalogs: true
I20240506 19:56:46.310194 13244 tablet_bootstrap.cc:492] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:46.313805 13249 tablet_bootstrap.cc:654] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.316757 13244 ts_tablet_manager.cc:1367] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.175s user 0.006s sys 0.008s
I20240506 19:56:46.318663 13244 raft_consensus.cc:348] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.322127 13245 tablet_bootstrap.cc:492] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.322376 13245 ts_tablet_manager.cc:1367] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.103s user 0.016s sys 0.020s
I20240506 19:56:46.322993 13244 raft_consensus.cc:374] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.323212 13244 raft_consensus.cc:729] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.323187 13245 raft_consensus.cc:348] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.323479 13245 raft_consensus.cc:374] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.323578 13245 raft_consensus.cc:729] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.323858 13245 consensus_queue.cc:260] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.323999 13244 consensus_queue.cc:260] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.324676 13245 ts_tablet_manager.cc:1398] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:46.325186 13245 tablet_bootstrap.cc:492] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.326972 13244 ts_tablet_manager.cc:1398] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:46.328490 13245 tablet_bootstrap.cc:654] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.328821 13244 tablet_bootstrap.cc:492] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:46.330596 13244 tablet_bootstrap.cc:654] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.333854 13245 tablet_bootstrap.cc:492] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.334512 13245 ts_tablet_manager.cc:1367] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240506 19:56:46.335474 13245 raft_consensus.cc:348] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.336845 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d57249f30a44dab3384092ec20b515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.337038 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet 028055b9df1f4667bd2e54909690e327 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.344928 13864 raft_consensus.cc:482] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.345091 13864 raft_consensus.cc:504] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.345754 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:46.345883 13864 leader_election.cc:288] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.345993 12513 catalog_manager.cc:5484] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:46.346241 13245 raft_consensus.cc:374] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.346328 13245 raft_consensus.cc:729] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.346554 13245 consensus_queue.cc:260] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.347107 13245 ts_tablet_manager.cc:1398] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20240506 19:56:46.347424 13245 tablet_bootstrap.cc:492] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.347527 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef03fcaa05d04ebd9f262f43ff7c02e4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.348111 12518 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578] alter complete (version 3)
I20240506 19:56:46.348248 13243 ts_tablet_manager.cc:1888] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.348346 13243 log.cc:1196] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d6e54543d1544b1c8cff3e890e787337
I20240506 19:56:46.348664 13243 ts_tablet_manager.cc:1909] T d6e54543d1544b1c8cff3e890e787337 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:46.348636 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e415d116a2364144bd571ef2ea61d225" } new_table_name: "default.Table_7d21a9e7790049a286bd3224ce33ad10" modify_external_catalogs: true
I20240506 19:56:46.348872 13245 tablet_bootstrap.cc:654] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.349038 13878 raft_consensus.cc:504] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.349757 13878 leader_election.cc:288] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.351325 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 88905d81f91d45d29c83a7b5df55f6f7 (DEFAULT_TABLE table=default.Table_6788948824bc43638d1e10322195e7e1 [id=4448b2536e3649c78cd6f6334852a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.351768 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88905d81f91d45d29c83a7b5df55f6f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:46.358949 12921 leader_election.cc:341] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.359795 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 42151ea2c1214c149d7dcac9a2c447aa
I20240506 19:56:46.360391 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 88905d81f91d45d29c83a7b5df55f6f7 (DEFAULT_TABLE table=default.Table_6788948824bc43638d1e10322195e7e1 [id=4448b2536e3649c78cd6f6334852a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.360519 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1750c4dd8cdc40509c553b63d166c466
I20240506 19:56:46.360795 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88905d81f91d45d29c83a7b5df55f6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.360852 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6b45eff5b94c4accac825233c6fa5303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.361316 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2db006c642ce4db78c5b7355300e9567 consensus state has no leader
W20240506 19:56:46.361634 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b45eff5b94c4accac825233c6fa5303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.362141 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a20016b56be94248905e0e8d6a35100f
W20240506 19:56:46.358372 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:56:46.362323 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 19:56:46.362479 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240506 19:56:46.362746 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 028055b9df1f4667bd2e54909690e327
I20240506 19:56:46.363171 12519 catalog_manager.cc:6554] ReplaceTablet: tablet 028055b9df1f4667bd2e54909690e327 deleted and replaced by tablet 71bfcd6f0f3c4a9597fef1461ec0efbe
I20240506 19:56:46.364333 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 42151ea2c1214c149d7dcac9a2c447aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.364545 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 1750c4dd8cdc40509c553b63d166c466 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.364713 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet a20016b56be94248905e0e8d6a35100f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.364862 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 028055b9df1f4667bd2e54909690e327 with delete_type TABLET_DATA_DELETED (replaced by tablet 71bfcd6f0f3c4a9597fef1461ec0efbe) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.365410 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 1750c4dd8cdc40509c553b63d166c466 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.365617 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 42151ea2c1214c149d7dcac9a2c447aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.366030 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:46.366119 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:46.366237 12986 raft_consensus.cc:3037] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.366756 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet a20016b56be94248905e0e8d6a35100f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.366966 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 028055b9df1f4667bd2e54909690e327 with delete_type TABLET_DATA_DELETED (replaced by tablet 71bfcd6f0f3c4a9597fef1461ec0efbe) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.367802 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet a20016b56be94248905e0e8d6a35100f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.367993 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 028055b9df1f4667bd2e54909690e327 with delete_type TABLET_DATA_DELETED (replaced by tablet 71bfcd6f0f3c4a9597fef1461ec0efbe) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:46.368191 13058 leader_election.cc:341] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.368562 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef03fcaa05d04ebd9f262f43ff7c02e4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.368769 13122 raft_consensus.cc:2454] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:46.368875 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 1750c4dd8cdc40509c553b63d166c466 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.369364 13864 raft_consensus.cc:482] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.369450 12922 leader_election.cc:302] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.369535 13864 raft_consensus.cc:504] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.369908 13911 raft_consensus.cc:2781] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.370004 13911 raft_consensus.cc:482] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.370108 13911 raft_consensus.cc:3037] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.370687 13864 leader_election.cc:288] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.371392 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c09e3bf7046ccba6ed25b08e3460e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.372090 12921 leader_election.cc:341] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.372594 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c09e3bf7046ccba6ed25b08e3460e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.372799 13122 raft_consensus.cc:2454] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:46.373631 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 42151ea2c1214c149d7dcac9a2c447aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.373907 13243 tablet_replica.cc:330] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:46.374138 13243 raft_consensus.cc:2229] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.374140 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d6e54543d1544b1c8cff3e890e787337 (table default.Table_77ba2f5ca8614d1ebb987a154041901c [id=3045c42f00ac4ff19e0354945092c6d1]) successfully deleted
I20240506 19:56:46.374316 13243 raft_consensus.cc:2258] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.374761 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a2e061a5a97646cc9b14ddf741c5b88e
I20240506 19:56:46.374897 13243 ts_tablet_manager.cc:1875] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.375945 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet a2e061a5a97646cc9b14ddf741c5b88e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.376653 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet a2e061a5a97646cc9b14ddf741c5b88e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.377724 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet a2e061a5a97646cc9b14ddf741c5b88e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.377745 12922 leader_election.cc:302] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.378163 13864 raft_consensus.cc:2781] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.378273 13864 raft_consensus.cc:482] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.378357 13864 raft_consensus.cc:3037] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.379671 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 449eeb67b602425bb40caa69b6a78a5a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.381927 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 17da01620b8b41008421b977455e0c9a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.382040 13245 tablet_bootstrap.cc:492] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.382246 13245 ts_tablet_manager.cc:1367] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.007s
I20240506 19:56:46.383143 13245 raft_consensus.cc:348] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.383493 13245 raft_consensus.cc:374] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.383592 13245 raft_consensus.cc:729] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.383880 13245 consensus_queue.cc:260] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.385067 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bb04d06a2b084079ac6357415b124752
I20240506 19:56:46.386996 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet cd6245e4adce4038ad6f988877d75045
I20240506 19:56:46.387451 12514 catalog_manager.cc:6554] ReplaceTablet: tablet cd6245e4adce4038ad6f988877d75045 deleted and replaced by tablet 996eb97639464e0bb976f2b1ecbbb73b
I20240506 19:56:46.389333 13245 ts_tablet_manager.cc:1398] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:46.389570 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6245e4adce4038ad6f988877d75045 with delete_type TABLET_DATA_DELETED (replaced by tablet 996eb97639464e0bb976f2b1ecbbb73b) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.389883 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet bb04d06a2b084079ac6357415b124752 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.390568 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6245e4adce4038ad6f988877d75045 with delete_type TABLET_DATA_DELETED (replaced by tablet 996eb97639464e0bb976f2b1ecbbb73b) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:46.390767 13885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 19:56:46.390944 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:56:46.391067 13885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 19:56:46.391130 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5809556edb740dfb542aae70194e631 consensus state has no leader
W20240506 19:56:46.391223 13885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 19:56:46.392329 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet bb04d06a2b084079ac6357415b124752 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.394117 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet bb04d06a2b084079ac6357415b124752 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.394315 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6245e4adce4038ad6f988877d75045 with delete_type TABLET_DATA_DELETED (replaced by tablet 996eb97639464e0bb976f2b1ecbbb73b) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.394398 13244 tablet_bootstrap.cc:492] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:46.394600 13244 ts_tablet_manager.cc:1367] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.002s
I20240506 19:56:46.394670 13245 tablet_bootstrap.cc:492] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.395509 13244 raft_consensus.cc:348] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.395740 13244 raft_consensus.cc:374] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.395844 13244 raft_consensus.cc:729] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.396080 13244 consensus_queue.cc:260] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.396900 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 88905d81f91d45d29c83a7b5df55f6f7 (DEFAULT_TABLE table=default.Table_6788948824bc43638d1e10322195e7e1 [id=4448b2536e3649c78cd6f6334852a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.397603 13245 tablet_bootstrap.cc:654] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.398711 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb29db8a2340a786edc11e3e370077 (DEFAULT_TABLE table=default.Table_31d45289fb564acaa4d5f2e45801473d [id=63774e46bd574bc6b5e0ad87a84c1a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.399915 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 88dba275893f4696a5318ec21f0b2477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.400141 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 104337875517445f862d82ece3a84889 with delete_type TABLET_DATA_DELETED (replaced by tablet 176c61208b1c4518820a829d9f5c0a6f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.401011 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb29db8a2340a786edc11e3e370077 (DEFAULT_TABLE table=default.Table_31d45289fb564acaa4d5f2e45801473d [id=63774e46bd574bc6b5e0ad87a84c1a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.405611 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 449eeb67b602425bb40caa69b6a78a5a
I20240506 19:56:46.406134 12517 catalog_manager.cc:6554] ReplaceTablet: tablet 449eeb67b602425bb40caa69b6a78a5a deleted and replaced by tablet d6edff8e7e6945aa926aa1486cbbc2bc
I20240506 19:56:46.407034 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 449eeb67b602425bb40caa69b6a78a5a with delete_type TABLET_DATA_DELETED (replaced by tablet d6edff8e7e6945aa926aa1486cbbc2bc) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.407604 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 449eeb67b602425bb40caa69b6a78a5a with delete_type TABLET_DATA_DELETED (replaced by tablet d6edff8e7e6945aa926aa1486cbbc2bc) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.407974 13244 ts_tablet_manager.cc:1398] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240506 19:56:46.409564 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9296bb1104b4cbe279eecd3683796) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.410460 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 449eeb67b602425bb40caa69b6a78a5a with delete_type TABLET_DATA_DELETED (replaced by tablet d6edff8e7e6945aa926aa1486cbbc2bc) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.409036 13244 tablet_bootstrap.cc:492] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:46.414880 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 1488328703024ae8983add9e4d6afc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:34 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.415829 13885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.252:36867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.48.252
--webserver_port=37261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:56:46.417614 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:56:46.417644 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 82b51eaa68fa45829891fc5b759110aa (DEFAULT_TABLE table=default.Table_a40f3b8780d740ee995e313e40e3b6fa [id=6557800ff36d45959d95996a473ff371]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.417881 13244 tablet_bootstrap.cc:654] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.418646 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb29db8a2340a786edc11e3e370077 (DEFAULT_TABLE table=default.Table_31d45289fb564acaa4d5f2e45801473d [id=63774e46bd574bc6b5e0ad87a84c1a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.418838 13245 tablet_bootstrap.cc:492] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.419028 13245 ts_tablet_manager.cc:1367] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.000s
I20240506 19:56:46.419814 13245 raft_consensus.cc:348] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.420074 13245 raft_consensus.cc:374] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.420176 13245 raft_consensus.cc:729] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.420482 13245 consensus_queue.cc:260] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.420620 13885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 19:56:46.422722 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 82b51eaa68fa45829891fc5b759110aa (DEFAULT_TABLE table=default.Table_a40f3b8780d740ee995e313e40e3b6fa [id=6557800ff36d45959d95996a473ff371]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.424105 13249 tablet_bootstrap.cc:492] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.424289 13249 ts_tablet_manager.cc:1367] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.145s user 0.003s sys 0.000s
I20240506 19:56:46.424427 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 7037cab8dce1415ca11badeb693d9bec (DEFAULT_TABLE table=default.Table_e1bc856560494ef0ac2a3d693b45a7c0 [id=3af64f2f2b324a43add1c324664a45c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.425099 13249 raft_consensus.cc:348] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.425460 13249 raft_consensus.cc:374] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.425395 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 82b51eaa68fa45829891fc5b759110aa (DEFAULT_TABLE table=default.Table_a40f3b8780d740ee995e313e40e3b6fa [id=6557800ff36d45959d95996a473ff371]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.425561 13249 raft_consensus.cc:729] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.426626 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 7037cab8dce1415ca11badeb693d9bec (DEFAULT_TABLE table=default.Table_e1bc856560494ef0ac2a3d693b45a7c0 [id=3af64f2f2b324a43add1c324664a45c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.427502 13249 consensus_queue.cc:260] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.429714 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 9227f74e85914cafabad913dc6dfba04 (DEFAULT_TABLE table=default.Table_3b6103041f344349ac34c8f87e87adae [id=48d078ee57f347408b5d548f448de1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.430953 13249 ts_tablet_manager.cc:1398] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240506 19:56:46.431571 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 7037cab8dce1415ca11badeb693d9bec (DEFAULT_TABLE table=default.Table_e1bc856560494ef0ac2a3d693b45a7c0 [id=3af64f2f2b324a43add1c324664a45c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.433504 13245 ts_tablet_manager.cc:1398] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240506 19:56:46.435470 13249 tablet_bootstrap.cc:492] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.437748 13249 tablet_bootstrap.cc:654] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.439301 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet da41ef03182445f1946584c2bf015139 with delete_type TABLET_DATA_DELETED (replaced by tablet 472c63d8995346d78633154a8f0d4c2b) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.440599 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_21676c5997464e2d8e94bb0efdfead41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.441409 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21676c5997464e2d8e94bb0efdfead41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.441778 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_41ecc96260e940ca86684d10d102665d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.442338 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41ecc96260e940ca86684d10d102665d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.445637 13124 raft_consensus.cc:2454] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:46.446092 12986 raft_consensus.cc:2454] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:46.446218 13246 ts_tablet_manager.cc:1888] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.446381 13246 log.cc:1196] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:46.446688 13246 ts_tablet_manager.cc:1909] T cc8335f9308046279a25a4a95d8e0f65 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:46.447162 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7deb29db8a2340a786edc11e3e370077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.447250 13242 ts_tablet_manager.cc:1888] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.447402 13242 log.cc:1196] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/5516e8aad4e8474692b5b6c40bc483c1
I20240506 19:56:46.447764 13242 ts_tablet_manager.cc:1909] T 5516e8aad4e8474692b5b6c40bc483c1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:46.448114 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88905d81f91d45d29c83a7b5df55f6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.448661 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7deb29db8a2340a786edc11e3e370077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.449066 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b51eaa68fa45829891fc5b759110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.449667 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b51eaa68fa45829891fc5b759110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.450243 13057 leader_election.cc:302] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.453236 13941 raft_consensus.cc:2781] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:46.453397 13941 raft_consensus.cc:686] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:46.453653 13941 consensus_queue.cc:237] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.456036 13245 tablet_bootstrap.cc:492] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.455991 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e66674b291e942a4b42aeb308799018f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.456614 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e66674b291e942a4b42aeb308799018f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.458894 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_30f6a4c5ddac420ca7fafd3366c7cd17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.459753 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30f6a4c5ddac420ca7fafd3366c7cd17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.460695 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 9227f74e85914cafabad913dc6dfba04 (DEFAULT_TABLE table=default.Table_3b6103041f344349ac34c8f87e87adae [id=48d078ee57f347408b5d548f448de1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.461086 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9227f74e85914cafabad913dc6dfba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.461469 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7037cab8dce1415ca11badeb693d9bec. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:46.461499 13985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:46.462675 13944 raft_consensus.cc:482] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.462934 13944 raft_consensus.cc:504] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.463985 13944 leader_election.cc:288] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.464825 12781 leader_election.cc:341] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 4f91888e88234cbdbd00f48dc1574653
I20240506 19:56:46.465538 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7037cab8dce1415ca11badeb693d9bec. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:46.469944 12783 leader_election.cc:341] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 4f91888e88234cbdbd00f48dc1574653
I20240506 19:56:46.470124 12783 leader_election.cc:302] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.472887 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 196b339d0bc54426b69a60fe59f1af00 (DEFAULT_TABLE table=default.Table_460654e013224b0b862d4c4663839cc1 [id=7e9605522cd24b86adeec813cb0d7697]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.476764 12961 tablet_service.cc:1467] Processing CreateTablet for tablet a2e061a5a97646cc9b14ddf741c5b88e (DEFAULT_TABLE table=default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 [id=1880f76f4434494090340781bfef83ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.477619 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a3a24c46afbb4a6b88e13ace0af3c39b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.478114 12965 tablet_service.cc:1467] Processing CreateTablet for tablet d1ebd7b994584b2bab6d27f493deab92 (DEFAULT_TABLE table=default.Table_1707b02f8b93477eaa71e7e6e47da51c [id=034d601564f94db89efafdf7d3b35f05]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:46.478258 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3a24c46afbb4a6b88e13ace0af3c39b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.478996 12949 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 (DEFAULT_TABLE table=default.Table_c1ceccbbb3344105a1314370d915f2cc [id=ad369c0b344f481d8ca9d29a19a8a1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.479876 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 196b339d0bc54426b69a60fe59f1af00 (DEFAULT_TABLE table=default.Table_460654e013224b0b862d4c4663839cc1 [id=7e9605522cd24b86adeec813cb0d7697]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.480270 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196b339d0bc54426b69a60fe59f1af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.486477 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 196b339d0bc54426b69a60fe59f1af00 (DEFAULT_TABLE table=default.Table_460654e013224b0b862d4c4663839cc1 [id=7e9605522cd24b86adeec813cb0d7697]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.486960 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196b339d0bc54426b69a60fe59f1af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.487908 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 221e3b5120494336822036588cf65456 (DEFAULT_TABLE table=default.Table_79eff2000c124513b48aeefee23bb9a3 [id=8b305bf0e98344d1a63fe0c84e5ee0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.488766 12952 tablet_service.cc:1467] Processing CreateTablet for tablet a512993561e849239eca5c212f19af3f (DEFAULT_TABLE table=default.Table_730c6ad3babb49aaba5d81f48381daec [id=47243ad822944b879acaef4663ee6dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.489766 12819 tablet_service.cc:1467] Processing CreateTablet for tablet a2e061a5a97646cc9b14ddf741c5b88e (DEFAULT_TABLE table=default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 [id=1880f76f4434494090340781bfef83ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.490146 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e061a5a97646cc9b14ddf741c5b88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.493062 13245 tablet_bootstrap.cc:654] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.493420 12814 tablet_service.cc:1467] Processing CreateTablet for tablet d1ebd7b994584b2bab6d27f493deab92 (DEFAULT_TABLE table=default.Table_1707b02f8b93477eaa71e7e6e47da51c [id=034d601564f94db89efafdf7d3b35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.493865 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ebd7b994584b2bab6d27f493deab92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.496532 12824 tablet_service.cc:1467] Processing CreateTablet for tablet a512993561e849239eca5c212f19af3f (DEFAULT_TABLE table=default.Table_730c6ad3babb49aaba5d81f48381daec [id=47243ad822944b879acaef4663ee6dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.496953 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a512993561e849239eca5c212f19af3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.497493 13245 tablet_bootstrap.cc:492] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.497613 13245 ts_tablet_manager.cc:1367] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.001s
I20240506 19:56:46.497591 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 (DEFAULT_TABLE table=default.Table_c1ceccbbb3344105a1314370d915f2cc [id=ad369c0b344f481d8ca9d29a19a8a1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.497951 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8e331dc24e4d06bf81c8a4b7d70156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.498334 13245 raft_consensus.cc:348] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.498512 13245 raft_consensus.cc:374] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.498555 13245 raft_consensus.cc:729] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.498760 13245 consensus_queue.cc:260] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.499212 13245 ts_tablet_manager.cc:1398] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:46.499477 13245 tablet_bootstrap.cc:492] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.500897 12950 tablet_service.cc:1467] Processing CreateTablet for tablet cb4945e5757c401cacb39037459e2b57 (DEFAULT_TABLE table=default.Table_ae36658079f748608197f569b0133127 [id=f301f42c7ec249c482779c6ac18a73a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.500979 13245 tablet_bootstrap.cc:654] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.502730 12820 tablet_service.cc:1467] Processing CreateTablet for tablet cb4945e5757c401cacb39037459e2b57 (DEFAULT_TABLE table=default.Table_ae36658079f748608197f569b0133127 [id=f301f42c7ec249c482779c6ac18a73a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.503136 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4945e5757c401cacb39037459e2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.506031 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 603) took 1425 ms. Trace:
I20240506 19:56:46.506132 13246 rpcz_store.cc:276] 0506 19:56:45.081074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.098562 (+ 17488us) service_pool.cc:225] Handling call
0506 19:56:46.260080 (+1161518us) ts_tablet_manager.cc:1120] Deleting tablet cc8335f9308046279a25a4a95d8e0f65
0506 19:56:46.260092 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.353459 (+ 93367us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.446073 (+ 92614us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.506003 (+ 59930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161274}
I20240506 19:56:46.506609 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 616) took 1116 ms. Trace:
I20240506 19:56:46.506594 12817 tablet_service.cc:1467] Processing CreateTablet for tablet bd3130bc41084bfaabaaecc0985e378d (DEFAULT_TABLE table=default.Table_a5d3e2fa72f2440eae3408f5b092bd7c [id=4905fa51dba84e71a032ba8b924c3751]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.506675 13246 rpcz_store.cc:276] 0506 19:56:45.389647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.401074 (+ 11427us) service_pool.cc:225] Handling call
0506 19:56:46.506476 (+1105402us) ts_tablet_manager.cc:1120] Deleting tablet f68f57002f8144caaaf37c2838f98e1f
0506 19:56:46.506485 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.506591 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105070}
I20240506 19:56:46.507007 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3130bc41084bfaabaaecc0985e378d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.506981 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 615) took 1117 ms. Trace:
I20240506 19:56:46.507046 13246 rpcz_store.cc:276] 0506 19:56:45.389481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.403035 (+ 13554us) service_pool.cc:225] Handling call
0506 19:56:46.506869 (+1103834us) ts_tablet_manager.cc:1120] Deleting tablet da06187c83b94d03ae4ebb67a2873cf0
0506 19:56:46.506878 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.506962 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103613}
I20240506 19:56:46.507351 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 617) took 1101 ms. Trace:
I20240506 19:56:46.507426 13246 rpcz_store.cc:276] 0506 19:56:45.405534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.406528 (+ 994us) service_pool.cc:225] Handling call
0506 19:56:46.507256 (+1100728us) ts_tablet_manager.cc:1120] Deleting tablet 99f4861a01404663870ce91978c8af4f
0506 19:56:46.507264 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.507336 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100498}
I20240506 19:56:46.507728 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 619) took 1097 ms. Trace:
I20240506 19:56:46.507797 13246 rpcz_store.cc:276] 0506 19:56:45.409979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.410121 (+ 142us) service_pool.cc:225] Handling call
0506 19:56:46.507611 (+1097490us) ts_tablet_manager.cc:1120] Deleting tablet 55d9f32618b64ffa92b5a868081c36a6
0506 19:56:46.507621 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.507714 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097279}
I20240506 19:56:46.508013 13246 tablet_replica.cc:330] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:46.509001 13246 raft_consensus.cc:2229] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.509159 13246 raft_consensus.cc:2258] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.510143 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 468) took 3827 ms. Trace:
I20240506 19:56:46.510268 13242 rpcz_store.cc:276] 0506 19:56:42.682291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.682516 (+ 225us) service_pool.cc:225] Handling call
0506 19:56:46.243114 (+3560598us) ts_tablet_manager.cc:1120] Deleting tablet 5516e8aad4e8474692b5b6c40bc483c1
0506 19:56:46.243123 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.350110 (+106987us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.447088 (+ 96978us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.510070 (+ 62982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3545851}
I20240506 19:56:46.510609 13242 tablet_replica.cc:330] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:46.510564 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 574dbbe8a9a94cd7944b652a68b77403 (DEFAULT_TABLE table=default.Table_7ae101bd10f242f28c19bd685449fce8 [id=b244f6a76bb04367b625ec4d7a249006]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.510813 13242 raft_consensus.cc:2229] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.511049 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574dbbe8a9a94cd7944b652a68b77403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.511958 12922 leader_election.cc:302] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:46.512275 12922 raft_consensus.cc:2670] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:46.513370 13089 tablet_service.cc:1467] Processing CreateTablet for tablet d1ebd7b994584b2bab6d27f493deab92 (DEFAULT_TABLE table=default.Table_1707b02f8b93477eaa71e7e6e47da51c [id=034d601564f94db89efafdf7d3b35f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.513571 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 (DEFAULT_TABLE table=default.Table_c1ceccbbb3344105a1314370d915f2cc [id=ad369c0b344f481d8ca9d29a19a8a1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.513795 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ebd7b994584b2bab6d27f493deab92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.513684 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 574dbbe8a9a94cd7944b652a68b77403 (DEFAULT_TABLE table=default.Table_7ae101bd10f242f28c19bd685449fce8 [id=b244f6a76bb04367b625ec4d7a249006]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.514104 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 221e3b5120494336822036588cf65456 (DEFAULT_TABLE table=default.Table_79eff2000c124513b48aeefee23bb9a3 [id=8b305bf0e98344d1a63fe0c84e5ee0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.514451 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221e3b5120494336822036588cf65456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.517683 12964 tablet_service.cc:1467] Processing CreateTablet for tablet bd3130bc41084bfaabaaecc0985e378d (DEFAULT_TABLE table=default.Table_a5d3e2fa72f2440eae3408f5b092bd7c [id=4905fa51dba84e71a032ba8b924c3751]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.518219 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8e331dc24e4d06bf81c8a4b7d70156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.518656 13944 raft_consensus.cc:2726] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.519815 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9227f74e85914cafabad913dc6dfba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.520434 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574dbbe8a9a94cd7944b652a68b77403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.521471 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7deb29db8a2340a786edc11e3e370077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.521883 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b51eaa68fa45829891fc5b759110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.522279 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7037cab8dce1415ca11badeb693d9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.522689 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196b339d0bc54426b69a60fe59f1af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.523110 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e061a5a97646cc9b14ddf741c5b88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.523500 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ebd7b994584b2bab6d27f493deab92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.523912 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8e331dc24e4d06bf81c8a4b7d70156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.524324 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221e3b5120494336822036588cf65456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.525724 13086 tablet_service.cc:1467] Processing CreateTablet for tablet a2e061a5a97646cc9b14ddf741c5b88e (DEFAULT_TABLE table=default.Table_e396d97dfe14406aaed9b0a4a7d0ddf5 [id=1880f76f4434494090340781bfef83ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.526094 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 5516e8aad4e8474692b5b6c40bc483c1 (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
I20240506 19:56:46.526173 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2e061a5a97646cc9b14ddf741c5b88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.511023 13242 raft_consensus.cc:2258] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.527478 13242 ts_tablet_manager.cc:1875] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.535378 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a512993561e849239eca5c212f19af3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.536679 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3130bc41084bfaabaaecc0985e378d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.537528 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4945e5757c401cacb39037459e2b57. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:46.545156 12921 consensus_peers.cc:485] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:46.545276 13246 ts_tablet_manager.cc:1875] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.545751 13911 raft_consensus.cc:504] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.548207 13911 leader_election.cc:288] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.552706 13243 ts_tablet_manager.cc:1888] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.554073 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef03fcaa05d04ebd9f262f43ff7c02e4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:46.547142 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef03fcaa05d04ebd9f262f43ff7c02e4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:46.557011 13124 raft_consensus.cc:3037] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.587033 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 9227f74e85914cafabad913dc6dfba04 (DEFAULT_TABLE table=default.Table_3b6103041f344349ac34c8f87e87adae [id=48d078ee57f347408b5d548f448de1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.590487 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 221e3b5120494336822036588cf65456 (DEFAULT_TABLE table=default.Table_79eff2000c124513b48aeefee23bb9a3 [id=8b305bf0e98344d1a63fe0c84e5ee0fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:46.554834 12921 leader_election.cc:341] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:46.600427 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 45574.3us
I20240506 19:56:46.556457 13864 raft_consensus.cc:504] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.601570 13864 leader_election.cc:288] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.602416 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c09e3bf7046ccba6ed25b08e3460e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:46.603032 12921 leader_election.cc:341] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.603672 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c09e3bf7046ccba6ed25b08e3460e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:46.603893 13122 raft_consensus.cc:3037] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.605137 13243 log.cc:1196] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/e17197adc6ca4593ac3fc494548a6e67
I20240506 19:56:46.605638 13085 tablet_service.cc:1467] Processing CreateTablet for tablet bd3130bc41084bfaabaaecc0985e378d (DEFAULT_TABLE table=default.Table_a5d3e2fa72f2440eae3408f5b092bd7c [id=4905fa51dba84e71a032ba8b924c3751]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.606702 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3130bc41084bfaabaaecc0985e378d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.609236 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 574dbbe8a9a94cd7944b652a68b77403 (DEFAULT_TABLE table=default.Table_7ae101bd10f242f28c19bd685449fce8 [id=b244f6a76bb04367b625ec4d7a249006]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.609627 13243 ts_tablet_manager.cc:1909] T e17197adc6ca4593ac3fc494548a6e67 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:46.609678 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574dbbe8a9a94cd7944b652a68b77403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.613178 13097 tablet_service.cc:1467] Processing CreateTablet for tablet cb4945e5757c401cacb39037459e2b57 (DEFAULT_TABLE table=default.Table_ae36658079f748608197f569b0133127 [id=f301f42c7ec249c482779c6ac18a73a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.616117 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4945e5757c401cacb39037459e2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.619665 13104 tablet_service.cc:1467] Processing CreateTablet for tablet a512993561e849239eca5c212f19af3f (DEFAULT_TABLE table=default.Table_730c6ad3babb49aaba5d81f48381daec [id=47243ad822944b879acaef4663ee6dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:46.620093 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a512993561e849239eca5c212f19af3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.626016 13878 raft_consensus.cc:482] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.626204 13878 raft_consensus.cc:504] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.593679 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet cc8335f9308046279a25a4a95d8e0f65 (table default.Table_df1733520c4a4ae6a99a78f194d6cf06 [id=64a093df5c7143668da7d3e91eccba2a]) successfully deleted
I20240506 19:56:46.626945 13878 leader_election.cc:288] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:46.626971 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f68f57002f8144caaaf37c2838f98e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f68f57002f8144caaaf37c2838f98e1f already in progress: creating tablet
W20240506 19:56:46.627226 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet da06187c83b94d03ae4ebb67a2873cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da06187c83b94d03ae4ebb67a2873cf0 already in progress: creating tablet
W20240506 19:56:46.627432 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 99f4861a01404663870ce91978c8af4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99f4861a01404663870ce91978c8af4f already in progress: creating tablet
W20240506 19:56:46.627633 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 55d9f32618b64ffa92b5a868081c36a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55d9f32618b64ffa92b5a868081c36a6 already in progress: creating tablet
I20240506 19:56:46.627568 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104337875517445f862d82ece3a84889" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:46.628300 13057 leader_election.cc:341] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.628785 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104337875517445f862d82ece3a84889" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.628789 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b0ca455fa29e479f80da85793bcc6414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.629448 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0ca455fa29e479f80da85793bcc6414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:46.629611 13058 leader_election.cc:341] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.629720 13058 leader_election.cc:302] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.630173 13878 raft_consensus.cc:2726] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.633747 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_46d49a82b855486f994526543440ace7" } new_table_name: "default.Table_8ef666d2b87f4076a267d567a422a9c8" modify_external_catalogs: true
I20240506 19:56:46.636831 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bca67f775cc9457cb0bbcd9b659322e8" } new_table_name: "default.Table_9a6d6630a6d643548740709cfeb1deb0" modify_external_catalogs: true
W20240506 19:56:46.640908 13987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:56:46.641417 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7a0c06c663854f80a34fbac74d6564e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.642182 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a0c06c663854f80a34fbac74d6564e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.641440 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6788948824bc43638d1e10322195e7e1" } new_table_name: "default.Table_ee7c8d5a5ae641beb78728caf7fd6603" modify_external_catalogs: true
I20240506 19:56:46.647029 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fac42373406c413f9eb5907032a077f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.647591 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fac42373406c413f9eb5907032a077f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.655650 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c09c5cd32c53422ca015e46ffcb2cfe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.656328 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c09c5cd32c53422ca015e46ffcb2cfe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.660024 13944 raft_consensus.cc:482] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.660174 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3b6103041f344349ac34c8f87e87adae" } modify_external_catalogs: true
I20240506 19:56:46.660239 13944 raft_consensus.cc:504] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.660382 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3b6103041f344349ac34c8f87e87adae" } modify_external_catalogs: true
I20240506 19:56:46.661000 13944 leader_election.cc:288] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.662079 12783 leader_election.cc:341] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b529bb9362924caca399491e7d9e03cd
W20240506 19:56:46.663339 12781 leader_election.cc:341] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b529bb9362924caca399491e7d9e03cd
I20240506 19:56:46.663479 12781 leader_election.cc:302] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.664044 13944 raft_consensus.cc:2726] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.664163 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fd8d81f3e15442a4926979026de42b20" } modify_external_catalogs: true
I20240506 19:56:46.664352 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fd8d81f3e15442a4926979026de42b20" } modify_external_catalogs: true
I20240506 19:56:46.663877 13249 tablet_bootstrap.cc:492] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.668119 13124 raft_consensus.cc:2454] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:46.668278 13249 ts_tablet_manager.cc:1367] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.233s user 0.005s sys 0.000s
I20240506 19:56:46.668738 12922 leader_election.cc:302] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.670828 13911 raft_consensus.cc:2781] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:46.670965 13911 raft_consensus.cc:686] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:46.671243 13911 consensus_queue.cc:237] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.672385 13244 tablet_bootstrap.cc:492] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:46.672564 13244 ts_tablet_manager.cc:1367] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.264s user 0.004s sys 0.003s
I20240506 19:56:46.673378 13244 raft_consensus.cc:348] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.673740 13244 raft_consensus.cc:374] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.673938 13244 raft_consensus.cc:729] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.674405 13244 consensus_queue.cc:260] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.675246 13244 ts_tablet_manager.cc:1398] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:46.676802 13249 raft_consensus.cc:348] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.677219 13249 raft_consensus.cc:374] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.677404 13249 raft_consensus.cc:729] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.677868 13249 consensus_queue.cc:260] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.679203 13249 ts_tablet_manager.cc:1398] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240506 19:56:46.679896 13249 tablet_bootstrap.cc:492] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.681139 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 649) took 1016 ms. Trace:
I20240506 19:56:46.682408 13243 rpcz_store.cc:276] 0506 19:56:45.664877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.664925 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:46.373858 (+708933us) ts_tablet_manager.cc:1120] Deleting tablet e17197adc6ca4593ac3fc494548a6e67
0506 19:56:46.373867 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.519669 (+145802us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.552551 (+ 32882us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.681114 (+128563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":708714}
I20240506 19:56:46.683182 13243 tablet_replica.cc:330] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:46.683542 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet e17197adc6ca4593ac3fc494548a6e67 (table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]) successfully deleted
I20240506 19:56:46.684165 13243 raft_consensus.cc:2229] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:46.685115 13244 tablet_bootstrap.cc:492] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:46.688612 13243 pending_rounds.cc:62] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:46.688740 13243 pending_rounds.cc:68] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744064054497280 op_type: NO_OP noop_request { }
I20240506 19:56:46.688939 13243 raft_consensus.cc:2866] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:46.694324 13941 raft_consensus.cc:482] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.694891 13941 raft_consensus.cc:504] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.696173 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2c9f4baf55412d9bc00818a00c652d" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.697008 13058 leader_election.cc:341] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.697566 13243 raft_consensus.cc:2258] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.698597 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2c9f4baf55412d9bc00818a00c652d" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:46.698881 12986 raft_consensus.cc:2454] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:46.699319 13057 leader_election.cc:302] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.699633 13878 raft_consensus.cc:2781] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.699757 13878 raft_consensus.cc:482] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.699843 13878 raft_consensus.cc:3037] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:46.700299 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88905d81f91d45d29c83a7b5df55f6f7 consensus state has no leader
I20240506 19:56:46.700822 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3612dd9500444861ac651d75a8b106a8" } new_table_name: "default.Table_31c22580ea6a414da4a17078ed4f88ef" modify_external_catalogs: true
W20240506 19:56:46.701619 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a5217c257ff46e48c23563e496b6f08 consensus state has no leader
I20240506 19:56:46.702040 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4674aa2f59294605bc41516bbae12158" } modify_external_catalogs: true
I20240506 19:56:46.702296 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4674aa2f59294605bc41516bbae12158" } modify_external_catalogs: true
I20240506 19:56:46.703230 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_200204bb39c0432081665c45b4aa9eae" } modify_external_catalogs: true
I20240506 19:56:46.703474 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_200204bb39c0432081665c45b4aa9eae" } modify_external_catalogs: true
W20240506 19:56:46.704098 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f63e8e12fd3b453a96413f1fc769fb9f consensus state has no leader
I20240506 19:56:46.704480 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b41fdc2ddabb43b6abd4d24fecd7447a" } modify_external_catalogs: true
I20240506 19:56:46.704708 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b41fdc2ddabb43b6abd4d24fecd7447a" } modify_external_catalogs: true
I20240506 19:56:46.700266 13941 leader_election.cc:288] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.705665 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_57c82ed0d1044d20b72a0e9341b299c5" } new_table_name: "default.Table_5de7f6e4f9f5452291f85ec5db486f81" modify_external_catalogs: true
I20240506 19:56:46.706569 13243 ts_tablet_manager.cc:1875] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.706874 13941 raft_consensus.cc:482] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.707185 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9fed2f12eda54be0879b1e9c33ca3e31
I20240506 19:56:46.707610 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9227f74e85914cafabad913dc6dfba04
I20240506 19:56:46.708295 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.709700 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.711270 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_641a8c0a4cee47a18707ea343732c1d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.711499 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c56ef126a4ca48b6a379a6771bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:46.711807 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_641a8c0a4cee47a18707ea343732c1d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.707576 13941 raft_consensus.cc:504] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.713768 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0551b9fce765497080cf8cad54f4dc9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.714319 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0551b9fce765497080cf8cad54f4dc9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.715469 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 9227f74e85914cafabad913dc6dfba04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.715813 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fed2f12eda54be0879b1e9c33ca3e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.716784 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fed2f12eda54be0879b1e9c33ca3e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.717412 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9227f74e85914cafabad913dc6dfba04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.717860 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b3ed0b56aa478abb44347afca8fcf6" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.718533 13058 leader_election.cc:341] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.719457 13941 leader_election.cc:288] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.724133 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9227f74e85914cafabad913dc6dfba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.724726 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221e3b5120494336822036588cf65456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:46.728730 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b3ed0b56aa478abb44347afca8fcf6" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:46.729379 13057 leader_election.cc:341] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.729532 13057 leader_election.cc:302] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.730196 13996 raft_consensus.cc:2726] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.730580 13249 tablet_bootstrap.cc:654] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.734593 13244 tablet_bootstrap.cc:654] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.737644 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 9227f74e85914cafabad913dc6dfba04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.738415 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7e80a457061942da9dbca15a6b063d8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.738917 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fed2f12eda54be0879b1e9c33ca3e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:46.739078 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e80a457061942da9dbca15a6b063d8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.743690 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9f9b64059bcb4cb0b125b84b6f163ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.744359 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f9b64059bcb4cb0b125b84b6f163ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.751410 13122 raft_consensus.cc:2454] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:46.757624 12922 leader_election.cc:302] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.759128 13911 raft_consensus.cc:2781] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:46.759307 13911 raft_consensus.cc:686] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:46.759558 13911 consensus_queue.cc:237] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.761739 13944 raft_consensus.cc:482] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.762008 13944 raft_consensus.cc:504] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.762846 13944 leader_election.cc:288] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.771224 12781 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 30d819d1237f4798b4847fdf378cbc62
W20240506 19:56:46.775583 12783 leader_election.cc:341] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 30d819d1237f4798b4847fdf378cbc62
I20240506 19:56:46.775825 12783 leader_election.cc:302] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.776571 13242 ts_tablet_manager.cc:1888] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.776773 13242 log.cc:1196] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/27b24fa4663244a88dac9ea99fe38f3a
I20240506 19:56:46.777431 13944 raft_consensus.cc:2726] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.777974 13242 ts_tablet_manager.cc:1909] T 27b24fa4663244a88dac9ea99fe38f3a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:46.778425 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 469) took 4093 ms. Trace:
I20240506 19:56:46.778570 13242 rpcz_store.cc:276] 0506 19:56:42.685025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.689438 (+ 4413us) service_pool.cc:225] Handling call
0506 19:56:46.510519 (+3821081us) ts_tablet_manager.cc:1120] Deleting tablet 27b24fa4663244a88dac9ea99fe38f3a
0506 19:56:46.510529 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.649470 (+138941us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.775364 (+125894us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.778368 (+ 3004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3812917}
I20240506 19:56:46.779294 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 27b24fa4663244a88dac9ea99fe38f3a (table default.Table_1f7a4e714f4a4412a5843504c95b87d7 [id=9844f952118d4225a5a7489527a6a703]) successfully deleted
I20240506 19:56:46.779958 13242 tablet_replica.cc:330] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:46.780349 13242 raft_consensus.cc:2229] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.780668 13242 raft_consensus.cc:2258] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.781536 13242 ts_tablet_manager.cc:1875] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.786805 13878 raft_consensus.cc:504] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.787860 13878 leader_election.cc:288] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.788717 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2c9f4baf55412d9bc00818a00c652d" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:46.788933 12986 raft_consensus.cc:3037] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.788954 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2c9f4baf55412d9bc00818a00c652d" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:46.789984 13245 tablet_bootstrap.cc:492] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.791672 13885 server_base.cc:1034] running on GCE node
I20240506 19:56:46.792601 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa
W20240506 19:56:46.792696 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7763959136c44674888916321bd6da24 consensus state has no leader
I20240506 19:56:46.795176 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b5789735f41642769315e305be4cc6ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.796491 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f45e9fa2cb945d2a04fa716d1ab33b4 consensus state has no leader
I20240506 19:56:46.796933 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f04d735a43fb445c9e751f02de612410"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.797499 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f04d735a43fb445c9e751f02de612410 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.799907 13944 raft_consensus.cc:482] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.800112 13944 raft_consensus.cc:504] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.800264 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:46.800395 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b89fb24d94042e1bfa772265a991680 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:46.795800 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5789735f41642769315e305be4cc6ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.800941 13944 leader_election.cc:288] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.801568 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:46.802250 12781 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:46.803267 13244 tablet_bootstrap.cc:492] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:46.803431 13244 ts_tablet_manager.cc:1367] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.000s
I20240506 19:56:46.804859 13244 raft_consensus.cc:348] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.805114 13244 raft_consensus.cc:374] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.805472 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:46.805933 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c56ef126a4ca48b6a379a6771bc1c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.807137 13244 raft_consensus.cc:729] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.807968 13244 consensus_queue.cc:260] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.808615 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 88dba275893f4696a5318ec21f0b2477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.809557 13244 ts_tablet_manager.cc:1398] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240506 19:56:46.809859 13244 tablet_bootstrap.cc:492] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:46.811592 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:46.813315 13244 tablet_bootstrap.cc:654] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.814473 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 104337875517445f862d82ece3a84889 with delete_type TABLET_DATA_DELETED (replaced by tablet 176c61208b1c4518820a829d9f5c0a6f) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:46.816391 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 19:56:46.817339 13058 leader_election.cc:341] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.817827 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:56:46.817386 13246 ts_tablet_manager.cc:1888] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.818019 13246 log.cc:1196] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:46.818378 13246 ts_tablet_manager.cc:1909] T 20c1590e36b4471d8e72f4b07410ee0f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:46.819829 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 618) took 1410 ms. Trace:
I20240506 19:56:46.819957 13246 rpcz_store.cc:276] 0506 19:56:45.409663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.410188 (+ 525us) service_pool.cc:225] Handling call
0506 19:56:46.507975 (+1097787us) ts_tablet_manager.cc:1120] Deleting tablet 20c1590e36b4471d8e72f4b07410ee0f
0506 19:56:46.507983 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.689253 (+181270us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.791368 (+102115us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.819777 (+ 28409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097588}
I20240506 19:56:46.820691 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 620) took 1390 ms. Trace:
I20240506 19:56:46.820845 13246 rpcz_store.cc:276] 0506 19:56:45.430677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.430799 (+ 122us) service_pool.cc:225] Handling call
0506 19:56:46.820490 (+1389691us) ts_tablet_manager.cc:1120] Deleting tablet cd57563cc6c140c184d1725a2da6a475
0506 19:56:46.820505 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.820649 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1389312}
I20240506 19:56:46.821278 13246 tablet_replica.cc:330] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:46.817266 13245 ts_tablet_manager.cc:1367] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.291s user 0.035s sys 0.056s
I20240506 19:56:46.821712 13944 raft_consensus.cc:482] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.821659 13122 raft_consensus.cc:2454] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:46.821877 13944 raft_consensus.cc:504] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.822690 13944 leader_election.cc:288] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.824258 12781 leader_election.cc:341] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 72a3bed0eb234e21b890242e9d23a5b4
W20240506 19:56:46.824609 12783 leader_election.cc:341] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:46.824740 12783 leader_election.cc:302] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.824981 13944 raft_consensus.cc:2726] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.825138 12783 leader_election.cc:302] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745
I20240506 19:56:46.828152 13944 raft_consensus.cc:2781] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.828261 13944 raft_consensus.cc:482] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.828361 13944 raft_consensus.cc:3037] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.829954 13249 tablet_bootstrap.cc:492] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.830113 13249 ts_tablet_manager.cc:1367] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.150s user 0.003s sys 0.000s
I20240506 19:56:46.830659 13246 raft_consensus.cc:2229] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.830964 13249 raft_consensus.cc:348] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.831230 13249 raft_consensus.cc:374] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.831290 13249 raft_consensus.cc:729] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.831696 13246 raft_consensus.cc:2258] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.832254 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 20c1590e36b4471d8e72f4b07410ee0f (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:46.823024 13245 raft_consensus.cc:348] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.832935 13245 raft_consensus.cc:374] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.833006 13245 raft_consensus.cc:729] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:46.832612 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet cd57563cc6c140c184d1725a2da6a475 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd57563cc6c140c184d1725a2da6a475 already in progress: creating tablet
I20240506 19:56:46.833436 13245 consensus_queue.cc:260] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.831482 13249 consensus_queue.cc:260] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.835626 13245 ts_tablet_manager.cc:1398] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240506 19:56:46.838285 13245 tablet_bootstrap.cc:492] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.839359 13249 ts_tablet_manager.cc:1398] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240506 19:56:46.840368 13249 tablet_bootstrap.cc:492] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.848692 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet da41ef03182445f1946584c2bf015139 with delete_type TABLET_DATA_DELETED (replaced by tablet 472c63d8995346d78633154a8f0d4c2b) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.848783 13249 tablet_bootstrap.cc:654] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.852661 13245 tablet_bootstrap.cc:654] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.866354 12986 raft_consensus.cc:2454] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:46.870781 13243 ts_tablet_manager.cc:1888] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:46.870939 13243 log.cc:1196] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1372be71e1d141ed89e58cdb806431a4
I20240506 19:56:46.871066 13249 tablet_bootstrap.cc:492] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.871205 13249 ts_tablet_manager.cc:1367] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20240506 19:56:46.871322 13243 ts_tablet_manager.cc:1909] T 1372be71e1d141ed89e58cdb806431a4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:46.871858 13242 ts_tablet_manager.cc:1888] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:46.872015 13242 log.cc:1196] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/0ba2365d00044c32a06f9017a207adf7
I20240506 19:56:46.872167 13246 ts_tablet_manager.cc:1875] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.872459 13242 ts_tablet_manager.cc:1909] T 0ba2365d00044c32a06f9017a207adf7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:46.872823 13249 raft_consensus.cc:348] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.873019 13249 raft_consensus.cc:374] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.873064 13249 raft_consensus.cc:729] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.873266 13249 consensus_queue.cc:260] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.873989 13249 ts_tablet_manager.cc:1398] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 19:56:46.874260 13249 tablet_bootstrap.cc:492] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.878173 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d53f3daaa2949a2b6883435f031dc37 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.882000 13057 leader_election.cc:302] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.883010 13878 raft_consensus.cc:2781] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:46.883208 13878 raft_consensus.cc:686] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:46.883584 13878 consensus_queue.cc:237] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.886811 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 234e4f3439e84b5fbb2f7bab1dbd0822 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.887449 13245 tablet_bootstrap.cc:492] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.887694 13245 ts_tablet_manager.cc:1367] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.004s
I20240506 19:56:46.890200 13944 raft_consensus.cc:504] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.890440 13245 raft_consensus.cc:348] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.890661 13245 raft_consensus.cc:374] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.890779 13245 raft_consensus.cc:729] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:46.892603 12781 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:46.891006 13245 consensus_queue.cc:260] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.894109 13944 leader_election.cc:288] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.893075 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c56ef126a4ca48b6a379a6771bc1c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:46.894917 12783 leader_election.cc:341] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:46.895030 12783 leader_election.cc:302] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.895597 13245 ts_tablet_manager.cc:1398] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:56:46.896003 13944 raft_consensus.cc:482] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.896332 13944 raft_consensus.cc:504] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.896922 13245 tablet_bootstrap.cc:492] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.897755 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:46.898285 14004 raft_consensus.cc:2726] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.899690 13249 tablet_bootstrap.cc:654] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.899899 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41faccd54f3e49ddb48ea5234f6cdab5" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.900554 12783 leader_election.cc:302] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.902331 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9296bb1104b4cbe279eecd3683796) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.905769 13245 tablet_bootstrap.cc:654] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.914623 12513 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 2 got 0
I20240506 19:56:46.915800 13996 raft_consensus.cc:482] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.916013 13996 raft_consensus.cc:504] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.916777 13996 leader_election.cc:288] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:46.917660 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3e7df09f347bfa5cad4de34dbda13" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:46.917646 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3e7df09f347bfa5cad4de34dbda13" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:46.918277 13057 leader_election.cc:341] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:46.918453 13058 leader_election.cc:341] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.918570 13058 leader_election.cc:302] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.918563 14004 raft_consensus.cc:2726] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.919324 13996 raft_consensus.cc:2726] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.920331 13249 tablet_bootstrap.cc:492] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.920533 13249 ts_tablet_manager.cc:1367] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.002s
I20240506 19:56:46.920776 13948 raft_consensus.cc:482] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.920903 13948 raft_consensus.cc:504] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.921551 13948 leader_election.cc:288] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.922212 13245 tablet_bootstrap.cc:492] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:46.922258 12921 consensus_peers.cc:485] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:46.922466 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88dba275893f4696a5318ec21f0b2477" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.922614 13249 raft_consensus.cc:348] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.922789 13249 raft_consensus.cc:374] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.922843 13249 raft_consensus.cc:729] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.923048 13249 consensus_queue.cc:260] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:46.923192 12921 leader_election.cc:341] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.922526 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88dba275893f4696a5318ec21f0b2477" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.923418 13122 raft_consensus.cc:2454] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:46.924513 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a9407b54478a4b51b62ed049b31e30bc" } new_table_name: "default.Table_feaa9d23df4841deb38df4084445bd09" modify_external_catalogs: true
I20240506 19:56:46.924727 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1707b02f8b93477eaa71e7e6e47da51c" } new_table_name: "default.Table_b86c5299ed814806a264d42742da4f36" modify_external_catalogs: true
I20240506 19:56:46.926120 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_11d89ae7590b466eacd8a986b0d180a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.926646 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11d89ae7590b466eacd8a986b0d180a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.927877 12922 leader_election.cc:302] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.928334 13245 ts_tablet_manager.cc:1367] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20240506 19:56:46.929133 13245 raft_consensus.cc:348] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.930001 13948 raft_consensus.cc:2781] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.930080 13948 raft_consensus.cc:482] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.930133 13948 raft_consensus.cc:3037] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.933884 13249 ts_tablet_manager.cc:1398] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240506 19:56:46.934089 13245 raft_consensus.cc:374] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.934250 13245 raft_consensus.cc:729] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.934597 13249 tablet_bootstrap.cc:492] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:46.935086 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0eb81c2cf9b549a28daebf834d26613c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.898618 13944 leader_election.cc:288] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.935384 13245 consensus_queue.cc:260] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.937036 13245 ts_tablet_manager.cc:1398] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240506 19:56:46.937893 13245 tablet_bootstrap.cc:492] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.938871 13249 tablet_bootstrap.cc:654] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:46.940029 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1ebd7b994584b2bab6d27f493deab92 consensus state has no leader
W20240506 19:56:46.940364 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e17d0a261734d45866525b35de7cba3 consensus state has no leader
I20240506 19:56:46.941205 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 472c63d8995346d78633154a8f0d4c2b from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:46.943162 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0eb81c2cf9b549a28daebf834d26613c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.946534 13911 raft_consensus.cc:482] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.946718 13911 raft_consensus.cc:504] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:46.947362 13911 leader_election.cc:288] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.948127 13245 tablet_bootstrap.cc:654] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:46.948627 13944 raft_consensus.cc:482] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:46.948813 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f0a5ff6d430282256e9bddc186b9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:46.948841 13944 raft_consensus.cc:504] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:46.949409 12921 leader_election.cc:341] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.948172 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f0a5ff6d430282256e9bddc186b9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:46.949729 13122 raft_consensus.cc:2454] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:46.950424 12922 leader_election.cc:302] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:46.950850 13911 raft_consensus.cc:2781] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:46.950956 13911 raft_consensus.cc:482] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:46.951037 13911 raft_consensus.cc:3037] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.952519 13864 consensus_queue.cc:1035] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:46.957242 13948 raft_consensus.cc:504] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.958001 13948 leader_election.cc:288] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:46.958429 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 99f4861a01404663870ce91978c8af4f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:46.958755 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88dba275893f4696a5318ec21f0b2477" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:46.959112 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c895888dd3b4499a990be1065b5cdbc8" } new_table_name: "default.Table_4508525747cc417a8274f7f3e0a71753" modify_external_catalogs: true
W20240506 19:56:46.959457 12921 leader_election.cc:341] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:46.960091 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88dba275893f4696a5318ec21f0b2477" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:46.960294 13124 raft_consensus.cc:3037] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:46.963735 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 648) took 1298 ms. Trace:
I20240506 19:56:46.963833 13243 rpcz_store.cc:276] 0506 19:56:45.664772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.665184 (+ 412us) service_pool.cc:225] Handling call
0506 19:56:46.683128 (+1017944us) ts_tablet_manager.cc:1120] Deleting tablet 1372be71e1d141ed89e58cdb806431a4
0506 19:56:46.683137 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.781630 (+ 98493us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.870607 (+ 88977us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.963710 (+ 93103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017714}
I20240506 19:56:46.963970 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8b925b5a2690446181a8ccfc20c9f959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:46.964283 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 650) took 1297 ms. Trace:
I20240506 19:56:46.964349 13243 rpcz_store.cc:276] 0506 19:56:45.667129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.667187 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:46.964143 (+1296956us) ts_tablet_manager.cc:1120] Deleting tablet ac9ecd2e034e49e49772db7435da3c81
0506 19:56:46.964154 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.964265 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296721}
I20240506 19:56:46.964671 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 656) took 1023 ms. Trace:
W20240506 19:56:46.964761 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b925b5a2690446181a8ccfc20c9f959 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.964751 13243 rpcz_store.cc:276] 0506 19:56:45.941660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.941771 (+ 111us) service_pool.cc:225] Handling call
0506 19:56:46.964547 (+1022776us) ts_tablet_manager.cc:1120] Deleting tablet 796beb8e0a2c41d698bd4c05796263f1
0506 19:56:46.964556 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.964653 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022579}
I20240506 19:56:46.965056 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 657) took 1008 ms. Trace:
I20240506 19:56:46.965117 13243 rpcz_store.cc:276] 0506 19:56:45.956940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.976244 (+ 19304us) service_pool.cc:225] Handling call
0506 19:56:46.964940 (+988696us) ts_tablet_manager.cc:1120] Deleting tablet 159ea8e0a4ee47fea93eb4b36cdd2495
0506 19:56:46.964949 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.965040 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":988499}
W20240506 19:56:46.965713 12781 leader_election.cc:341] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 1488328703024ae8983add9e4d6afc8f
I20240506 19:56:46.966078 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1372be71e1d141ed89e58cdb806431a4 (table default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]) successfully deleted
I20240506 19:56:46.966161 13944 leader_election.cc:288] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:46.966526 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ac9ecd2e034e49e49772db7435da3c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9ecd2e034e49e49772db7435da3c81 already in progress: creating tablet
W20240506 19:56:46.966958 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 796beb8e0a2c41d698bd4c05796263f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 796beb8e0a2c41d698bd4c05796263f1 already in progress: creating tablet
W20240506 19:56:46.967275 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 159ea8e0a4ee47fea93eb4b36cdd2495 already in progress: creating tablet
I20240506 19:56:46.969691 13243 tablet_replica.cc:330] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:46.969950 13243 raft_consensus.cc:2229] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
W20240506 19:56:46.970428 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c0c7cae21c714148a37cab389d116e91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0c7cae21c714148a37cab389d116e91 already in progress: creating tablet
I20240506 19:56:46.970772 13243 pending_rounds.cc:62] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
W20240506 19:56:46.970762 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4be4e628b9364cccb525b8e1b8070286 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be4e628b9364cccb525b8e1b8070286 already in progress: creating tablet
I20240506 19:56:46.970824 13243 pending_rounds.cc:68] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744066090246144 op_type: NO_OP noop_request { }
I20240506 19:56:46.970925 13243 raft_consensus.cc:2866] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:46.970999 13243 raft_consensus.cc:2258] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:46.971014 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7bc30f4690b24c5ea8ab2f7a26122992 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bc30f4690b24c5ea8ab2f7a26122992 already in progress: creating tablet
W20240506 19:56:46.971199 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 42151ea2c1214c149d7dcac9a2c447aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42151ea2c1214c149d7dcac9a2c447aa already in progress: creating tablet
W20240506 19:56:46.971378 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1750c4dd8cdc40509c553b63d166c466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1750c4dd8cdc40509c553b63d166c466 already in progress: creating tablet
W20240506 19:56:46.971562 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a20016b56be94248905e0e8d6a35100f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20016b56be94248905e0e8d6a35100f already in progress: creating tablet
W20240506 19:56:46.971765 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 028055b9df1f4667bd2e54909690e327 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 028055b9df1f4667bd2e54909690e327 already in progress: creating tablet
W20240506 19:56:46.971930 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a2e061a5a97646cc9b14ddf741c5b88e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e061a5a97646cc9b14ddf741c5b88e already in progress: creating tablet
W20240506 19:56:46.972110 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet cd6245e4adce4038ad6f988877d75045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6245e4adce4038ad6f988877d75045 already in progress: creating tablet
W20240506 19:56:46.972271 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bb04d06a2b084079ac6357415b124752 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb04d06a2b084079ac6357415b124752 already in progress: creating tablet
I20240506 19:56:46.972252 13243 ts_tablet_manager.cc:1875] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:46.972486 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 449eeb67b602425bb40caa69b6a78a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 449eeb67b602425bb40caa69b6a78a5a already in progress: creating tablet
W20240506 19:56:46.973749 12783 leader_election.cc:341] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1488328703024ae8983add9e4d6afc8f
I20240506 19:56:46.973881 12783 leader_election.cc:302] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:46.977893 13944 raft_consensus.cc:2726] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:46.979308 13245 tablet_bootstrap.cc:492] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:46.979724 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7a0c06c663854f80a34fbac74d6564e2" } modify_external_catalogs: true
I20240506 19:56:46.979967 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7a0c06c663854f80a34fbac74d6564e2" } modify_external_catalogs: true
I20240506 19:56:46.981707 13245 ts_tablet_manager.cc:1367] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.004s
I20240506 19:56:46.982326 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4180588e9c5d43cd95909d0f1cb7a140" } modify_external_catalogs: true
I20240506 19:56:46.982586 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4180588e9c5d43cd95909d0f1cb7a140" } modify_external_catalogs: true
I20240506 19:56:46.983940 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 470) took 4285 ms. Trace:
I20240506 19:56:46.983644 13249 tablet_bootstrap.cc:492] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:46.984894 13249 ts_tablet_manager.cc:1367] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.005s
I20240506 19:56:46.985761 13249 raft_consensus.cc:348] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.985950 13249 raft_consensus.cc:374] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:46.985998 13249 raft_consensus.cc:729] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.986300 13245 raft_consensus.cc:348] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.986614 13242 rpcz_store.cc:276] 0506 19:56:42.698399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.698480 (+ 81us) service_pool.cc:225] Handling call
0506 19:56:46.779866 (+4081386us) ts_tablet_manager.cc:1120] Deleting tablet 0ba2365d00044c32a06f9017a207adf7
0506 19:56:46.779881 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.786268 (+ 6387us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.871696 (+ 85428us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:46.983915 (+112219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4080012}
I20240506 19:56:46.987090 13245 raft_consensus.cc:374] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:46.987193 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8a6ca6ea4fe400abbde9ec7f07cc79e consensus state has no leader
I20240506 19:56:46.987361 13245 raft_consensus.cc:729] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:46.987738 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b820628699774953a9a7b655edefe41c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:46.988651 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b820628699774953a9a7b655edefe41c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:46.988715 13245 consensus_queue.cc:260] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.988946 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 0ba2365d00044c32a06f9017a207adf7 (table default.Table_24bc90d17d404ca3884c13c02624e651 [id=7a329c37e7684bac8bf008b1c5e53e1c]) successfully deleted
I20240506 19:56:46.989408 13242 tablet_replica.cc:330] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:46.989874 13242 raft_consensus.cc:2229] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:46.990298 13242 raft_consensus.cc:2258] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:46.991066 13242 ts_tablet_manager.cc:1875] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:46.991771 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 01c32eba2031446ab419cac64068068e from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:46.992528 13249 consensus_queue.cc:260] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:46.995654 13249 ts_tablet_manager.cc:1398] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240506 19:56:46.994275 13245 ts_tablet_manager.cc:1398] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240506 19:56:46.998385 13245 tablet_bootstrap.cc:492] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:46.998592 13249 tablet_bootstrap.cc:492] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.000231 12514 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a2403ca4811c465f8d33d8fd84a36c37
I20240506 19:56:47.003862 13249 tablet_bootstrap.cc:654] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.004387 13245 tablet_bootstrap.cc:654] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.013593 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bf6a9df4bf204889a5a4737a2a61b964" } modify_external_catalogs: true
I20240506 19:56:47.013904 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bf6a9df4bf204889a5a4737a2a61b964" } modify_external_catalogs: true
I20240506 19:56:47.014745 13944 raft_consensus.cc:482] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.014902 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b86c5299ed814806a264d42742da4f36" } new_table_name: "default.Table_b261bf09f51646a0877f068d9eb2ab36" modify_external_catalogs: true
I20240506 19:56:47.015166 13944 raft_consensus.cc:504] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.015376 13245 tablet_bootstrap.cc:492] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.015707 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e13e6c9a0cd1442db0f3cb1b4c6cb5f4" } new_table_name: "default.Table_659a4d19b7d447438d293f59be74f51a" modify_external_catalogs: true
W20240506 19:56:47.016719 12781 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 8f66f1e46cce47639fe644fb4ed2e5c0
W20240506 19:56:47.018793 12783 leader_election.cc:341] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:47.018909 12783 leader_election.cc:302] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:47.019161 14004 raft_consensus.cc:2726] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:47.019373 13944 leader_election.cc:288] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.025660 13249 tablet_bootstrap.cc:492] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.025843 13249 ts_tablet_manager.cc:1367] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20240506 19:56:47.026640 13249 raft_consensus.cc:348] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.026922 13249 raft_consensus.cc:374] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.026994 13249 raft_consensus.cc:729] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.027307 13249 consensus_queue.cc:260] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.028023 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20240506 19:56:47.034122 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2db006c642ce4db78c5b7355300e9567 consensus state has no leader
I20240506 19:56:47.036619 13249 ts_tablet_manager.cc:1398] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240506 19:56:47.037055 13249 tablet_bootstrap.cc:492] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.040086 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ff86f5a848874ce7960fbcc9c79457fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.041478 13249 tablet_bootstrap.cc:654] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:47.042265 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff86f5a848874ce7960fbcc9c79457fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:47.044823 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1ebd7b994584b2bab6d27f493deab92 consensus state has no leader
I20240506 19:56:47.045835 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_351b532454104aebb7decad2864d0629"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.046864 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351b532454104aebb7decad2864d0629 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.052206 13244 tablet_bootstrap.cc:492] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:47.054989 13244 ts_tablet_manager.cc:1367] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.245s user 0.068s sys 0.026s
I20240506 19:56:47.055452 13249 tablet_bootstrap.cc:492] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.055604 13249 ts_tablet_manager.cc:1367] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.003s
I20240506 19:56:47.056210 13244 raft_consensus.cc:348] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.056270 13249 raft_consensus.cc:348] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.056459 13249 raft_consensus.cc:374] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.056514 13249 raft_consensus.cc:729] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.056555 13244 raft_consensus.cc:374] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.056706 13244 raft_consensus.cc:729] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.056958 13124 raft_consensus.cc:2454] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:47.056977 13911 raft_consensus.cc:504] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.057021 13244 consensus_queue.cc:260] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.057612 12922 leader_election.cc:302] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.057654 13911 leader_election.cc:288] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.058121 13864 raft_consensus.cc:2781] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:47.058224 13864 raft_consensus.cc:686] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:47.058254 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f0a5ff6d430282256e9bddc186b9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.058441 13864 consensus_queue.cc:237] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:47.058763 12921 leader_election.cc:341] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.061084 13244 ts_tablet_manager.cc:1398] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:47.061442 13244 tablet_bootstrap.cc:492] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:47.062656 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f0a5ff6d430282256e9bddc186b9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:47.062856 13124 raft_consensus.cc:3037] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.064234 13246 ts_tablet_manager.cc:1888] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:47.064344 13246 log.cc:1196] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:47.064640 13246 ts_tablet_manager.cc:1909] T 618c56ef126a4ca48b6a379a6771bc1c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:47.067899 13885 hybrid_clock.cc:648] HybridClock initialized: now 1715025407067676 us; error 186 us; skew 500 ppm
I20240506 19:56:47.068150 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0da0ab98ad584f7fa1dbce910bd12569"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.068539 13885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240506 19:56:47.068866 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0da0ab98ad584f7fa1dbce910bd12569 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.056737 13249 consensus_queue.cc:260] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.071568 13249 ts_tablet_manager.cc:1398] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20240506 19:56:47.072957 13249 tablet_bootstrap.cc:492] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.074110 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet a36b4d51b100486c831a41a52d81fd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.074743 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5ac61897169e4a91bbe63d32fce65440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.075258 13249 tablet_bootstrap.cc:654] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:47.076524 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ac61897169e4a91bbe63d32fce65440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.085690 13244 tablet_bootstrap.cc:654] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.088024 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6f20f6b912c249479dbf461093583817"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.092531 13944 raft_consensus.cc:482] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.092720 13944 raft_consensus.cc:504] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:47.093039 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f20f6b912c249479dbf461093583817 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.093458 13944 leader_election.cc:288] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:47.094650 12783 leader_election.cc:341] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f3d57249f30a44dab3384092ec20b515
I20240506 19:56:47.097143 13245 ts_tablet_manager.cc:1367] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.099s user 0.018s sys 0.000s
W20240506 19:56:47.098271 12781 leader_election.cc:341] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f3d57249f30a44dab3384092ec20b515
I20240506 19:56:47.098412 12781 leader_election.cc:302] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:47.098948 13944 raft_consensus.cc:2726] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:47.100018 13245 raft_consensus.cc:348] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.100230 13245 raft_consensus.cc:374] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.100283 13245 raft_consensus.cc:729] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.101601 13245 consensus_queue.cc:260] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.105823 13245 ts_tablet_manager.cc:1398] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:47.110924 13864 consensus_queue.cc:1035] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:47.111286 13244 tablet_bootstrap.cc:492] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:47.111444 13244 ts_tablet_manager.cc:1367] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.000s
I20240506 19:56:47.112200 13244 raft_consensus.cc:348] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.112488 13244 raft_consensus.cc:374] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.112596 13244 raft_consensus.cc:729] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.112926 13244 consensus_queue.cc:260] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:47.113596 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 99f4861a01404663870ce91978c8af4f (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 99f4861a01404663870ce91978c8af4f consensus state has no leader
I20240506 19:56:47.114602 13245 tablet_bootstrap.cc:492] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.116330 13245 tablet_bootstrap.cc:654] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.120692 13249 tablet_bootstrap.cc:492] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.120863 13249 ts_tablet_manager.cc:1367] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.004s
I20240506 19:56:47.120750 13244 ts_tablet_manager.cc:1398] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:56:47.122011 13249 raft_consensus.cc:348] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.122227 13249 raft_consensus.cc:374] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.122272 13249 raft_consensus.cc:729] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.122493 13249 consensus_queue.cc:260] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.123183 13249 ts_tablet_manager.cc:1398] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:47.123472 13249 tablet_bootstrap.cc:492] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.125063 13249 tablet_bootstrap.cc:654] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.125599 13244 tablet_bootstrap.cc:492] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:47.127360 13244 tablet_bootstrap.cc:654] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.127151 12515 catalog_manager.cc:5484] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:47.127658 12515 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
I20240506 19:56:47.134614 13885 webserver.cc:460] Webserver started at http://127.12.48.252:37261/ using document root <none> and password file <none>
I20240506 19:56:47.135093 12954 tablet_service.cc:1467] Processing CreateTablet for tablet e99ce389141348aa9ed9c2c56279a24d (DEFAULT_TABLE table=default.Table_e66674b291e942a4b42aeb308799018f [id=6a7a686c1b3b441c998a25179c38e9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.135936 13885 fs_manager.cc:359] Metadata directory not provided
I20240506 19:56:47.136197 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:56:47.138012 13245 tablet_bootstrap.cc:492] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.138248 13245 ts_tablet_manager.cc:1367] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240506 19:56:47.139144 13245 raft_consensus.cc:348] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.139421 13245 raft_consensus.cc:374] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.139518 13245 raft_consensus.cc:729] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.139719 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 01c32eba2031446ab419cac64068068e
I20240506 19:56:47.141033 13245 consensus_queue.cc:260] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.141628 13245 ts_tablet_manager.cc:1398] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:47.141959 13245 tablet_bootstrap.cc:492] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.144145 12520 catalog_manager.cc:6554] ReplaceTablet: tablet 01c32eba2031446ab419cac64068068e deleted and replaced by tablet ce6483470c864697be6d9590ec95edfc
I20240506 19:56:47.144861 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c32eba2031446ab419cac64068068e with delete_type TABLET_DATA_DELETED (replaced by tablet ce6483470c864697be6d9590ec95edfc) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.146136 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c32eba2031446ab419cac64068068e with delete_type TABLET_DATA_DELETED (replaced by tablet ce6483470c864697be6d9590ec95edfc) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.148211 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:47.148536 13245 tablet_bootstrap.cc:654] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.149991 13249 tablet_bootstrap.cc:492] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.150223 13249 ts_tablet_manager.cc:1367] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
I20240506 19:56:47.150449 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.151737 13249 raft_consensus.cc:348] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.151962 13249 raft_consensus.cc:374] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.152010 13249 raft_consensus.cc:729] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.152541 13249 consensus_queue.cc:260] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.153111 13249 ts_tablet_manager.cc:1398] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:47.153141 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.153563 13249 tablet_bootstrap.cc:492] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.155097 13249 tablet_bootstrap.cc:654] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.151279 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c32eba2031446ab419cac64068068e with delete_type TABLET_DATA_DELETED (replaced by tablet ce6483470c864697be6d9590ec95edfc) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.157903 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99ce389141348aa9ed9c2c56279a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.157929 13243 ts_tablet_manager.cc:1888] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:47.157984 13244 tablet_bootstrap.cc:492] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:47.158035 13243 log.cc:1196] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/517c09e3bf7046ccba6ed25b08e3460e
I20240506 19:56:47.158143 13244 ts_tablet_manager.cc:1367] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.009s
I20240506 19:56:47.158391 13243 ts_tablet_manager.cc:1909] T 517c09e3bf7046ccba6ed25b08e3460e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:47.158777 13124 raft_consensus.cc:2454] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:47.158952 13244 raft_consensus.cc:348] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.159153 13244 raft_consensus.cc:374] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.159204 13244 raft_consensus.cc:729] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.159299 13242 ts_tablet_manager.cc:1888] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:47.159391 13242 log.cc:1196] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/30d819d1237f4798b4847fdf378cbc62
I20240506 19:56:47.159410 13244 consensus_queue.cc:260] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.159672 13242 ts_tablet_manager.cc:1909] T 30d819d1237f4798b4847fdf378cbc62 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:47.159999 13244 ts_tablet_manager.cc:1398] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:47.160275 13244 tablet_bootstrap.cc:492] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:47.162237 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 07755e37a20646a3b1a7b900d54a077c with delete_type TABLET_DATA_DELETED (replaced by tablet 034d6309de0740d0a5806b3847e334cb) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.164254 12922 leader_election.cc:302] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.164703 13911 raft_consensus.cc:482] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.164870 13911 raft_consensus.cc:504] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.165655 13911 leader_election.cc:288] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.165786 13911 raft_consensus.cc:2781] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:47.165901 13911 raft_consensus.cc:686] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:47.166035 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b3ed0b56aa478abb44347afca8fcf6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.166144 13911 consensus_queue.cc:237] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:47.166694 12921 leader_election.cc:341] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.168464 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b3ed0b56aa478abb44347afca8fcf6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.168679 13124 raft_consensus.cc:2454] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.169697 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:47.171475 12922 leader_election.cc:302] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.171922 13948 raft_consensus.cc:2781] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.171999 13948 raft_consensus.cc:482] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.172070 13948 raft_consensus.cc:3037] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.172736 13245 tablet_bootstrap.cc:492] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.172873 13245 ts_tablet_manager.cc:1367] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20240506 19:56:47.172842 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet ea25fb8e42234b82885b8fb2275d4925 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.173528 13245 raft_consensus.cc:348] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.173717 13245 raft_consensus.cc:374] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.173774 13245 raft_consensus.cc:729] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.174367 13245 consensus_queue.cc:260] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.174525 12809 tablet_service.cc:1467] Processing CreateTablet for tablet e99ce389141348aa9ed9c2c56279a24d (DEFAULT_TABLE table=default.Table_e66674b291e942a4b42aeb308799018f [id=6a7a686c1b3b441c998a25179c38e9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.153338 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 621) took 1646 ms. Trace:
I20240506 19:56:47.174933 13245 ts_tablet_manager.cc:1398] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:47.174973 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99ce389141348aa9ed9c2c56279a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.175215 13245 tablet_bootstrap.cc:492] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.176509 13245 tablet_bootstrap.cc:654] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.177987 13244 tablet_bootstrap.cc:654] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.174852 13246 rpcz_store.cc:276] 0506 19:56:45.506541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.506599 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:46.821220 (+1314621us) ts_tablet_manager.cc:1120] Deleting tablet 618c56ef126a4ca48b6a379a6771bc1c
0506 19:56:46.821227 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:46.831569 (+ 10342us) spinlock_profiling.cc:243] Waited 9.13 ms on lock 0x616000101554. stack: 00007f946cbb3756 00007f946cbb3510 00007f946bbe648d 0000000000955804 00000000009557cd 000000000095519a 00007f9478421a00 00007f94796fbb83 00007f947c18f4ec 00007f947c1c1844 00007f947c1a2b30 00007f947c1a28bd 0000000000a60bcc 00007f946cbf9b74 00007f946cbfd19b 00007f946cbfd018
0506 19:56:46.983337 (+151768us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.064108 (+ 80771us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.153281 (+ 89173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21008256,"tablet-delete.queue_time_us":1314364}
I20240506 19:56:47.180008 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 622) took 1654 ms. Trace:
I20240506 19:56:47.180058 12957 tablet_service.cc:1467] Processing CreateTablet for tablet b96dc180faef44d598edfa11fe65e64b (DEFAULT_TABLE table=default.Table_6b45eff5b94c4accac825233c6fa5303 [id=1bbd0af47d9a4dfb8f4d8e98117c888f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.180092 13246 rpcz_store.cc:276] 0506 19:56:45.525424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.525501 (+ 77us) service_pool.cc:225] Handling call
0506 19:56:47.179855 (+1654354us) ts_tablet_manager.cc:1120] Deleting tablet ea7e073296ce46e786362aaed96eb4dd
0506 19:56:47.179865 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.179987 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654027}
I20240506 19:56:47.180434 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96dc180faef44d598edfa11fe65e64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.180465 13246 tablet_replica.cc:330] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:47.180644 13246 raft_consensus.cc:2229] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:47.180809 13246 raft_consensus.cc:2258] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:47.181430 13246 ts_tablet_manager.cc:1875] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:47.181560 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 6a077a0ae1f4411c9fb2409311a9d9ba (DEFAULT_TABLE table=default.Table_6d330fca725b4de09fa2e8a21d67d676 [id=436998b2930242f8ad542a3a8ae6a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.181963 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a077a0ae1f4411c9fb2409311a9d9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.182706 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_12562fde5f794337b741599bd5fc9ba0" } new_table_name: "default.Table_81e0b435a0344362af8168947f3b1d56" modify_external_catalogs: true
I20240506 19:56:47.185242 12966 tablet_service.cc:1467] Processing CreateTablet for tablet a5b4375f10114c8085b76c7832275552 (DEFAULT_TABLE table=default.Table_b8b3b6c37e2d4fe9a43050bea37c80b7 [id=8e6a813495f84e8b964d966ee6673ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.185609 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b4375f10114c8085b76c7832275552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.188987 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 8d17df556f894102af1493104ac7cba0 (DEFAULT_TABLE table=default.Table_917d34debc2543bdb5c53f1cad733526 [id=eccaa60f1c9f49f0bc5176572b44ab76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.189559 12949 tablet_service.cc:1467] Processing CreateTablet for tablet f1a792bf9b774b8f8cc5491605f504ac (DEFAULT_TABLE table=default.Table_0027f2a42b1a494685c82f0862604186 [id=75b7767e5d0944f981db0bcf2e82e5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.192641 13911 raft_consensus.cc:482] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.192772 13911 raft_consensus.cc:504] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.193442 13911 leader_election.cc:288] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.194613 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2417a535329341d8bb9f8218770f890d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.194856 13124 raft_consensus.cc:2454] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.194826 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2417a535329341d8bb9f8218770f890d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.196151 12922 leader_election.cc:302] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:47.197006 12812 tablet_service.cc:1467] Processing CreateTablet for tablet c5751f7370d142a2914ac75b233e8c56 (DEFAULT_TABLE table=default.Table_21676c5997464e2d8e94bb0efdfead41 [id=106b2ba1a92643d4892c5e9197d81117]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.197469 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5751f7370d142a2914ac75b233e8c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.198380 13864 raft_consensus.cc:2781] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.198468 13864 raft_consensus.cc:482] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.198549 13864 raft_consensus.cc:3037] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.201332 12817 tablet_service.cc:1467] Processing CreateTablet for tablet b96dc180faef44d598edfa11fe65e64b (DEFAULT_TABLE table=default.Table_6b45eff5b94c4accac825233c6fa5303 [id=1bbd0af47d9a4dfb8f4d8e98117c888f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.201872 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96dc180faef44d598edfa11fe65e64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.205561 12820 tablet_service.cc:1467] Processing CreateTablet for tablet a5b4375f10114c8085b76c7832275552 (DEFAULT_TABLE table=default.Table_b8b3b6c37e2d4fe9a43050bea37c80b7 [id=8e6a813495f84e8b964d966ee6673ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.206051 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b4375f10114c8085b76c7832275552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.209575 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 8d17df556f894102af1493104ac7cba0 (DEFAULT_TABLE table=default.Table_917d34debc2543bdb5c53f1cad733526 [id=eccaa60f1c9f49f0bc5176572b44ab76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.209898 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d17df556f894102af1493104ac7cba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.213601 12824 tablet_service.cc:1467] Processing CreateTablet for tablet f1a792bf9b774b8f8cc5491605f504ac (DEFAULT_TABLE table=default.Table_0027f2a42b1a494685c82f0862604186 [id=75b7767e5d0944f981db0bcf2e82e5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.213939 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a792bf9b774b8f8cc5491605f504ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.205250 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 6a077a0ae1f4411c9fb2409311a9d9ba (DEFAULT_TABLE table=default.Table_6d330fca725b4de09fa2e8a21d67d676 [id=436998b2930242f8ad542a3a8ae6a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.218874 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a077a0ae1f4411c9fb2409311a9d9ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:47.222877 12921 leader_election.cc:341] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.222978 13885 fs_manager.cc:709] Time spent opening directory manager: real 0.079s user 0.004s sys 0.000s
I20240506 19:56:47.223372 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d17df556f894102af1493104ac7cba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.223640 13878 raft_consensus.cc:482] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.223774 13878 raft_consensus.cc:504] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.224457 13878 leader_election.cc:288] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.225168 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:47.225873 13058 leader_election.cc:341] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.226938 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a792bf9b774b8f8cc5491605f504ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.228327 13878 raft_consensus.cc:482] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.228510 13878 raft_consensus.cc:504] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.229349 13878 leader_election.cc:288] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.230180 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd57563cc6c140c184d1725a2da6a475" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.230803 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed with delete_type TABLET_DATA_DELETED (replaced by tablet a03f574aec984b83ab596e92a6f323fc) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:47.230929 13058 leader_election.cc:341] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.231209 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 996eb97639464e0bb976f2b1ecbbb73b (DEFAULT_TABLE table=default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.231583 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996eb97639464e0bb976f2b1ecbbb73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.233422 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 2417a535329341d8bb9f8218770f890d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.233795 12964 tablet_service.cc:1467] Processing CreateTablet for tablet adbaf70315ff442e996870d9b6ddcf48 (DEFAULT_TABLE table=default.Table_971fe431636342168d3237de7c0a271d [id=d38ac469f6724317ab026105379afa63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.234112 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbaf70315ff442e996870d9b6ddcf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.235225 13249 tablet_bootstrap.cc:492] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.235769 13249 ts_tablet_manager.cc:1367] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.012s
I20240506 19:56:47.236790 13249 raft_consensus.cc:348] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.237037 13249 raft_consensus.cc:374] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.237097 13249 raft_consensus.cc:729] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.237205 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 1980a68925be4d3da1078c78d32da866 (DEFAULT_TABLE table=default.Table_fb3b18d9d707453393cfb058b284bfa6 [id=b28d9eb2939949c69a11301afa33150d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.237480 13249 consensus_queue.cc:260] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.237944 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0c085734de48a0a51f43c4e5ddf77d (DEFAULT_TABLE table=default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20 [id=e9d56f7b210b48caac7ad8b211e7de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.238179 13249 ts_tablet_manager.cc:1398] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:47.238441 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 06234a1df4264ddd9abbcaf88292795f (DEFAULT_TABLE table=default.Table_94602d1502334dcb977668d7d42eebec [id=d2e2ea2858eb4b39b082df599d497112]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.238570 13249 tablet_bootstrap.cc:492] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.239145 12967 tablet_service.cc:1467] Processing CreateTablet for tablet bef5086682264414aedf3f1d50e88cc8 (DEFAULT_TABLE table=default.Table_2c27273f191c449d882ac1e69af5a4e0 [id=a27a2494edb24142adcb7725a6c9be42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.239600 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbcc40911034f319f489931015ca257 (DEFAULT_TABLE table=default.Table_a3a24c46afbb4a6b88e13ace0af3c39b [id=1560b01a73fc491d9d99019711972d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.240069 12960 tablet_service.cc:1467] Processing CreateTablet for tablet cdb4ae1443ca43e98eec0339326b29d0 (DEFAULT_TABLE table=default.Table_092635dc2dec452490c7710c0cc1a37e [id=4fab85c0d2a44f7d9e65f2f9be3b85f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.240481 12959 tablet_service.cc:1467] Processing CreateTablet for tablet d6edff8e7e6945aa926aa1486cbbc2bc (DEFAULT_TABLE table=default.Table_dccf1c1b22014892a25a3ea6da1ecf29 [id=f38b135c9bd24ddcac7c2d6b3f05a1d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.240923 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9b92795a7742caa2ee99ab7ca138bf (DEFAULT_TABLE table=default.Table_92e6b60423294d189484e6f67d5eb1ad [id=be8cafaac30841bd8f10581c1e5c0b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.241403 12956 tablet_service.cc:1467] Processing CreateTablet for tablet f424e713b96d43cdac6597c20bd5fc73 (DEFAULT_TABLE table=default.Table_c44481b32b644df187cba705ae9759c5 [id=f0ab9e171cb14e4a8b3b7d76ddbfcc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.242262 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 48769e0dd8864d3b85ee6a3ed1c03c94 (DEFAULT_TABLE table=default.Table_8c4e4e8dbeff40b6b23e5456dbb53729 [id=cb8610d949ac4b1f83b8646f3b7d2caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.242798 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 7446b0633a484cfba10f44a69dcc6969 (DEFAULT_TABLE table=default.Table_424aab7141bd4c80967b1104df604c32 [id=ed62c20271fe4b58be6d1d66643da8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.245393 13878 raft_consensus.cc:482] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.245534 13878 raft_consensus.cc:504] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.246227 13878 leader_election.cc:288] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.237646 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1980a68925be4d3da1078c78d32da866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.247145 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:47.247825 13058 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.249534 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6edff8e7e6945aa926aa1486cbbc2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.251964 13243 tablet_replica.cc:330] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:47.252211 13243 raft_consensus.cc:2229] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:47.252494 13243 pending_rounds.cc:62] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:47.252539 13243 pending_rounds.cc:68] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744065728434176 op_type: NO_OP noop_request { }
I20240506 19:56:47.252665 13243 raft_consensus.cc:2866] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:47.252753 13243 raft_consensus.cc:2258] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:47.253168 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 471) took 4553 ms. Trace:
I20240506 19:56:47.253245 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb4ae1443ca43e98eec0339326b29d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.253270 13242 rpcz_store.cc:276] 0506 19:56:42.700042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.700292 (+ 250us) service_pool.cc:225] Handling call
0506 19:56:46.989351 (+4289059us) ts_tablet_manager.cc:1120] Deleting tablet 30d819d1237f4798b4847fdf378cbc62
0506 19:56:46.989363 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.063771 (+ 74408us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.159157 (+ 95386us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.253136 (+ 93979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274078}
I20240506 19:56:47.253696 13243 ts_tablet_manager.cc:1875] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:47.253733 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 472) took 4553 ms. Trace:
I20240506 19:56:47.253803 13242 rpcz_store.cc:276] 0506 19:56:42.700766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.703401 (+ 2635us) service_pool.cc:225] Handling call
0506 19:56:47.253597 (+4550196us) ts_tablet_manager.cc:1120] Deleting tablet d9443cdbaa3c40299b66d216673c4b1a
0506 19:56:47.253608 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.253715 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4536434}
I20240506 19:56:47.254089 13242 tablet_replica.cc:330] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:47.254264 13242 raft_consensus.cc:2229] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:47.254408 13242 raft_consensus.cc:2258] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:47.254870 13242 ts_tablet_manager.cc:1875] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:47.256608 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9b92795a7742caa2ee99ab7ca138bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.259817 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbcc40911034f319f489931015ca257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.260239 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06234a1df4264ddd9abbcaf88292795f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.260586 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f424e713b96d43cdac6597c20bd5fc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.260962 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef5086682264414aedf3f1d50e88cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.261346 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0c085734de48a0a51f43c4e5ddf77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.261723 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48769e0dd8864d3b85ee6a3ed1c03c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.261888 13243 ts_tablet_manager.cc:1888] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:47.261999 13243 log.cc:1196] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:47.262746 13948 raft_consensus.cc:504] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.263447 13948 leader_election.cc:288] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.264937 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7446b0633a484cfba10f44a69dcc6969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.268138 12819 tablet_service.cc:1467] Processing CreateTablet for tablet adbaf70315ff442e996870d9b6ddcf48 (DEFAULT_TABLE table=default.Table_971fe431636342168d3237de7c0a271d [id=d38ac469f6724317ab026105379afa63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.268314 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 30d819d1237f4798b4847fdf378cbc62 (table default.Table_9d121b9e240b43cfbfea349bf45ef3a7 [id=332ee4f1670e4ddba0deb90d49e7b964]) successfully deleted
W20240506 19:56:47.268652 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d9443cdbaa3c40299b66d216673c4b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9443cdbaa3c40299b66d216673c4b1a already in progress: creating tablet
I20240506 19:56:47.268684 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 1980a68925be4d3da1078c78d32da866 (DEFAULT_TABLE table=default.Table_fb3b18d9d707453393cfb058b284bfa6 [id=b28d9eb2939949c69a11301afa33150d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.268684 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbcc40911034f319f489931015ca257 (DEFAULT_TABLE table=default.Table_a3a24c46afbb4a6b88e13ace0af3c39b [id=1560b01a73fc491d9d99019711972d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.269062 12810 tablet_service.cc:1467] Processing CreateTablet for tablet cdb4ae1443ca43e98eec0339326b29d0 (DEFAULT_TABLE table=default.Table_092635dc2dec452490c7710c0cc1a37e [id=4fab85c0d2a44f7d9e65f2f9be3b85f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.269230 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0c085734de48a0a51f43c4e5ddf77d (DEFAULT_TABLE table=default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20 [id=e9d56f7b210b48caac7ad8b211e7de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.269846 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 06234a1df4264ddd9abbcaf88292795f (DEFAULT_TABLE table=default.Table_94602d1502334dcb977668d7d42eebec [id=d2e2ea2858eb4b39b082df599d497112]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.270053 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 7446b0633a484cfba10f44a69dcc6969 (DEFAULT_TABLE table=default.Table_424aab7141bd4c80967b1104df604c32 [id=ed62c20271fe4b58be6d1d66643da8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.270315 12954 tablet_service.cc:1467] Processing CreateTablet for tablet c5751f7370d142a2914ac75b233e8c56 (DEFAULT_TABLE table=default.Table_21676c5997464e2d8e94bb0efdfead41 [id=106b2ba1a92643d4892c5e9197d81117]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.270490 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 48769e0dd8864d3b85ee6a3ed1c03c94 (DEFAULT_TABLE table=default.Table_8c4e4e8dbeff40b6b23e5456dbb53729 [id=cb8610d949ac4b1f83b8646f3b7d2caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.270710 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5751f7370d142a2914ac75b233e8c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.270936 12815 tablet_service.cc:1467] Processing CreateTablet for tablet bef5086682264414aedf3f1d50e88cc8 (DEFAULT_TABLE table=default.Table_2c27273f191c449d882ac1e69af5a4e0 [id=a27a2494edb24142adcb7725a6c9be42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.271347 12828 tablet_service.cc:1467] Processing CreateTablet for tablet d6edff8e7e6945aa926aa1486cbbc2bc (DEFAULT_TABLE table=default.Table_dccf1c1b22014892a25a3ea6da1ecf29 [id=f38b135c9bd24ddcac7c2d6b3f05a1d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.271720 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9b92795a7742caa2ee99ab7ca138bf (DEFAULT_TABLE table=default.Table_92e6b60423294d189484e6f67d5eb1ad [id=be8cafaac30841bd8f10581c1e5c0b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.272080 12825 tablet_service.cc:1467] Processing CreateTablet for tablet f424e713b96d43cdac6597c20bd5fc73 (DEFAULT_TABLE table=default.Table_c44481b32b644df187cba705ae9759c5 [id=f0ab9e171cb14e4a8b3b7d76ddbfcc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.272459 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 996eb97639464e0bb976f2b1ecbbb73b (DEFAULT_TABLE table=default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.273528 13243 ts_tablet_manager.cc:1909] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:47.274053 13249 tablet_bootstrap.cc:654] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.274794 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8c01d6060ec04ecea2651ad7fc8b9436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.275271 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b3ed0b56aa478abb44347afca8fcf6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:47.275452 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c01d6060ec04ecea2651ad7fc8b9436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:47.280514 12921 leader_election.cc:341] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.283217 13096 tablet_service.cc:1467] Processing CreateTablet for tablet f1a792bf9b774b8f8cc5491605f504ac (DEFAULT_TABLE table=default.Table_0027f2a42b1a494685c82f0862604186 [id=75b7767e5d0944f981db0bcf2e82e5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.283380 13103 tablet_service.cc:1467] Processing CreateTablet for tablet d6edff8e7e6945aa926aa1486cbbc2bc (DEFAULT_TABLE table=default.Table_dccf1c1b22014892a25a3ea6da1ecf29 [id=f38b135c9bd24ddcac7c2d6b3f05a1d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.283217 13091 tablet_service.cc:1467] Processing CreateTablet for tablet cdb4ae1443ca43e98eec0339326b29d0 (DEFAULT_TABLE table=default.Table_092635dc2dec452490c7710c0cc1a37e [id=4fab85c0d2a44f7d9e65f2f9be3b85f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.283908 13101 tablet_service.cc:1467] Processing CreateTablet for tablet f424e713b96d43cdac6597c20bd5fc73 (DEFAULT_TABLE table=default.Table_c44481b32b644df187cba705ae9759c5 [id=f0ab9e171cb14e4a8b3b7d76ddbfcc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.284194 13087 tablet_service.cc:1467] Processing CreateTablet for tablet e99ce389141348aa9ed9c2c56279a24d (DEFAULT_TABLE table=default.Table_e66674b291e942a4b42aeb308799018f [id=6a7a686c1b3b441c998a25179c38e9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.284473 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 1980a68925be4d3da1078c78d32da866 (DEFAULT_TABLE table=default.Table_fb3b18d9d707453393cfb058b284bfa6 [id=b28d9eb2939949c69a11301afa33150d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.284631 13086 tablet_service.cc:1467] Processing CreateTablet for tablet c5751f7370d142a2914ac75b233e8c56 (DEFAULT_TABLE table=default.Table_21676c5997464e2d8e94bb0efdfead41 [id=106b2ba1a92643d4892c5e9197d81117]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.284893 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0c085734de48a0a51f43c4e5ddf77d (DEFAULT_TABLE table=default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20 [id=e9d56f7b210b48caac7ad8b211e7de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.285105 13104 tablet_service.cc:1467] Processing CreateTablet for tablet b96dc180faef44d598edfa11fe65e64b (DEFAULT_TABLE table=default.Table_6b45eff5b94c4accac825233c6fa5303 [id=1bbd0af47d9a4dfb8f4d8e98117c888f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.285264 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 618c56ef126a4ca48b6a379a6771bc1c (table default.Table_b50238125ad14920bf820245406918c3 [id=7f61579ec04443a6992e572f3e52f980]) successfully deleted
I20240506 19:56:47.285326 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 06234a1df4264ddd9abbcaf88292795f (DEFAULT_TABLE table=default.Table_94602d1502334dcb977668d7d42eebec [id=d2e2ea2858eb4b39b082df599d497112]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:47.285559 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ea7e073296ce46e786362aaed96eb4dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7e073296ce46e786362aaed96eb4dd already in progress: creating tablet
I20240506 19:56:47.285568 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 6a077a0ae1f4411c9fb2409311a9d9ba (DEFAULT_TABLE table=default.Table_6d330fca725b4de09fa2e8a21d67d676 [id=436998b2930242f8ad542a3a8ae6a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.285759 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 7446b0633a484cfba10f44a69dcc6969 (DEFAULT_TABLE table=default.Table_424aab7141bd4c80967b1104df604c32 [id=ed62c20271fe4b58be6d1d66643da8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.283805 13092 tablet_service.cc:1467] Processing CreateTablet for tablet adbaf70315ff442e996870d9b6ddcf48 (DEFAULT_TABLE table=default.Table_971fe431636342168d3237de7c0a271d [id=d38ac469f6724317ab026105379afa63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.285984 13097 tablet_service.cc:1467] Processing CreateTablet for tablet a5b4375f10114c8085b76c7832275552 (DEFAULT_TABLE table=default.Table_b8b3b6c37e2d4fe9a43050bea37c80b7 [id=8e6a813495f84e8b964d966ee6673ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.286396 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 8d17df556f894102af1493104ac7cba0 (DEFAULT_TABLE table=default.Table_917d34debc2543bdb5c53f1cad733526 [id=eccaa60f1c9f49f0bc5176572b44ab76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.286783 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 48769e0dd8864d3b85ee6a3ed1c03c94 (DEFAULT_TABLE table=default.Table_8c4e4e8dbeff40b6b23e5456dbb53729 [id=cb8610d949ac4b1f83b8646f3b7d2caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.287312 13102 tablet_service.cc:1467] Processing CreateTablet for tablet bef5086682264414aedf3f1d50e88cc8 (DEFAULT_TABLE table=default.Table_2c27273f191c449d882ac1e69af5a4e0 [id=a27a2494edb24142adcb7725a6c9be42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.287506 13948 raft_consensus.cc:482] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.287642 13948 raft_consensus.cc:504] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.287745 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbcc40911034f319f489931015ca257 (DEFAULT_TABLE table=default.Table_a3a24c46afbb4a6b88e13ace0af3c39b [id=1560b01a73fc491d9d99019711972d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.288110 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9b92795a7742caa2ee99ab7ca138bf (DEFAULT_TABLE table=default.Table_92e6b60423294d189484e6f67d5eb1ad [id=be8cafaac30841bd8f10581c1e5c0b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.288326 13948 leader_election.cc:288] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.288529 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 996eb97639464e0bb976f2b1ecbbb73b (DEFAULT_TABLE table=default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.289270 12963 tablet_service.cc:1467] Processing CreateTablet for tablet b8f873037284401c89dc46182aa47830 (DEFAULT_TABLE table=default.Table_9a458e9734bc4cd78dd0eb34dc07783f [id=80227a440b0948cdae9918e92486e003]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.289741 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f873037284401c89dc46182aa47830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.290369 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.290982 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 517c09e3bf7046ccba6ed25b08e3460e (table default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 [id=4b18d70e1c8847de81b7742f55f56260]) successfully deleted
W20240506 19:56:47.291254 12921 leader_election.cc:341] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.299576 13249 tablet_bootstrap.cc:492] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.299752 13249 ts_tablet_manager.cc:1367] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.003s
I20240506 19:56:47.300730 13249 raft_consensus.cc:348] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.300979 13249 raft_consensus.cc:374] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.301044 13249 raft_consensus.cc:729] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.301321 13249 consensus_queue.cc:260] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.301859 13249 ts_tablet_manager.cc:1398] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:47.302184 13249 tablet_bootstrap.cc:492] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.303402 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6125a723a247a9b0bfaf55338e8a34 (DEFAULT_TABLE table=default.Table_30f6a4c5ddac420ca7fafd3366c7cd17 [id=077490f89fa147baa2713ab465729c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.303815 13249 tablet_bootstrap.cc:654] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.306849 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 19:56:47.309932 13911 raft_consensus.cc:482] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.310055 13911 raft_consensus.cc:504] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.310696 13911 leader_election.cc:288] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.312551 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6125a723a247a9b0bfaf55338e8a34 (DEFAULT_TABLE table=default.Table_30f6a4c5ddac420ca7fafd3366c7cd17 [id=077490f89fa147baa2713ab465729c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.312902 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6125a723a247a9b0bfaf55338e8a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.315294 13885 fs_manager.cc:725] Time spent opening block manager: real 0.044s user 0.005s sys 0.002s
I20240506 19:56:47.306282 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 8c47a3f8d6e3440f8ab2c0b1ab49118a (DEFAULT_TABLE table=default.Table_e71ba8bf0b874298b72cae27e8607c6c [id=e450843bf078436e83034537873af619]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.309615 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb398132c58489fb0a28492302e4a23 (DEFAULT_TABLE table=default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef [id=163ac8a58256479c87bbc87265058cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.315732 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1d53f3daaa2949a2b6883435f031dc37
I20240506 19:56:47.316246 12518 catalog_manager.cc:6554] ReplaceTablet: tablet 1d53f3daaa2949a2b6883435f031dc37 deleted and replaced by tablet c264c66cfec443c7a38677237ae2ba31
I20240506 19:56:47.316599 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8fe0b52e5f1640da9f2920e67e7e26bb" } new_table_name: "default.Table_c5b5c0c587d6447eb89d9677b41b4757" modify_external_catalogs: true
I20240506 19:56:47.312541 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.316298 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data,/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
uuid: "97f390a786034cddbff757779037f861"
format_stamp: "Formatted at 2024-05-06 19:56:29 on dist-test-slave-gjg4"
W20240506 19:56:47.319311 12921 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.315265 12824 tablet_service.cc:1467] Processing CreateTablet for tablet b8f873037284401c89dc46182aa47830 (DEFAULT_TABLE table=default.Table_9a458e9734bc4cd78dd0eb34dc07783f [id=80227a440b0948cdae9918e92486e003]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:47.320531 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 124012us
I20240506 19:56:47.322198 13245 tablet_bootstrap.cc:492] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.322427 13245 ts_tablet_manager.cc:1367] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.147s user 0.021s sys 0.017s
I20240506 19:56:47.322604 13244 tablet_bootstrap.cc:492] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:47.322763 13244 ts_tablet_manager.cc:1367] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.163s user 0.014s sys 0.026s
I20240506 19:56:47.323529 13244 raft_consensus.cc:348] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.323758 13244 raft_consensus.cc:374] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.323837 13244 raft_consensus.cc:729] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.323979 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:56:47.324014 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2c96f014293242be9640515697c0733b
I20240506 19:56:47.324064 13244 consensus_queue.cc:260] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.324901 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1b89fb24d94042e1bfa772265a991680
I20240506 19:56:47.325346 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 1b89fb24d94042e1bfa772265a991680 deleted and replaced by tablet dc9e8072e33347719b0f676ff6036e64
I20240506 19:56:47.325345 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:47.325549 12986 raft_consensus.cc:2454] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:47.325513 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:47.325703 12987 raft_consensus.cc:2454] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:47.322948 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b3ed0b56aa478abb44347afca8fcf6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:47.325981 13123 raft_consensus.cc:3037] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.326826 13245 raft_consensus.cc:348] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.326988 13245 raft_consensus.cc:374] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.327030 13245 raft_consensus.cc:729] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.327201 13245 consensus_queue.cc:260] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.327668 13245 ts_tablet_manager.cc:1398] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:47.328675 13249 tablet_bootstrap.cc:492] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:47.328797 13249 ts_tablet_manager.cc:1367] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240506 19:56:47.323540 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.329339 13122 raft_consensus.cc:2454] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.324751 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.329778 13124 raft_consensus.cc:2454] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.329737 13249 raft_consensus.cc:348] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.329905 13249 raft_consensus.cc:374] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.329965 13249 raft_consensus.cc:729] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.325345 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd57563cc6c140c184d1725a2da6a475" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:47.330322 12988 raft_consensus.cc:2454] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:47.330468 13245 tablet_bootstrap.cc:492] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.331315 13249 consensus_queue.cc:260] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.332844 13244 ts_tablet_manager.cc:1398] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240506 19:56:47.332985 13249 ts_tablet_manager.cc:1398] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:47.333799 13249 tablet_bootstrap.cc:492] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:47.334460 13245 tablet_bootstrap.cc:654] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.335376 13057 leader_election.cc:302] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.335605 13244 tablet_bootstrap.cc:492] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:47.335832 13878 raft_consensus.cc:2781] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.335911 13878 raft_consensus.cc:482] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.335971 13878 raft_consensus.cc:3037] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.336997 13057 leader_election.cc:302] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.337314 13244 tablet_bootstrap.cc:654] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.337327 13249 tablet_bootstrap.cc:654] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.337333 13996 raft_consensus.cc:2781] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.337402 13996 raft_consensus.cc:482] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.337477 13996 raft_consensus.cc:3037] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.337826 13057 leader_election.cc:302] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.345882 12922 leader_election.cc:302] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.346709 14034 raft_consensus.cc:2781] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.346791 14034 raft_consensus.cc:482] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.346853 14034 raft_consensus.cc:3037] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.348443 12922 leader_election.cc:302] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.349447 14033 raft_consensus.cc:2781] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.349530 14033 raft_consensus.cc:482] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.349598 14033 raft_consensus.cc:3037] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.349058 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 19:56:47.352036 13944 raft_consensus.cc:482] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.352188 13944 raft_consensus.cc:504] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.352852 13944 leader_election.cc:288] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:47.354120 12783 leader_election.cc:341] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2519c03738864f6ab5dcfab7bc3fc8aa
I20240506 19:56:47.362493 13864 raft_consensus.cc:504] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.363350 13864 leader_election.cc:288] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.372097 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe (DEFAULT_TABLE table=default.Table_51c8c97056294104afce55e84e26a550 [id=4a72c498da064f31a0c5437f8e0d1048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.372362 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 79592a4ecacb4324b2278e1b9a75496c (DEFAULT_TABLE table=default.Table_41ecc96260e940ca86684d10d102665d [id=c42c984f783c4bafaca702b9af8e42f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.373499 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 9466fd3447474ca4b6f99c3810e2454e (DEFAULT_TABLE table=default.Table_1f6bae23516f4e79b0662b14c8a8ea3a [id=19bea1ba9dbf4b2787c397c6152b8c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.376566 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 8c47a3f8d6e3440f8ab2c0b1ab49118a (DEFAULT_TABLE table=default.Table_e71ba8bf0b874298b72cae27e8607c6c [id=e450843bf078436e83034537873af619]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.377009 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c47a3f8d6e3440f8ab2c0b1ab49118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.378585 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ef03fcaa05d04ebd9f262f43ff7c02e4 (table default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]) successfully deleted
I20240506 19:56:47.378789 13243 tablet_replica.cc:330] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:47.379006 13243 raft_consensus.cc:2229] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:47.379245 13243 pending_rounds.cc:62] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
W20240506 19:56:47.379264 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9fed2f12eda54be0879b1e9c33ca3e31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fed2f12eda54be0879b1e9c33ca3e31 already in progress: creating tablet
I20240506 19:56:47.379297 13243 pending_rounds.cc:68] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744067313577984 op_type: NO_OP noop_request { }
I20240506 19:56:47.379422 13243 raft_consensus.cc:2866] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:47.379503 13243 raft_consensus.cc:2258] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:47.379539 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9227f74e85914cafabad913dc6dfba04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9227f74e85914cafabad913dc6dfba04 already in progress: creating tablet
W20240506 19:56:47.379765 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa already in progress: creating tablet
I20240506 19:56:47.380345 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2417a535329341d8bb9f8218770f890d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.381000 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb398132c58489fb0a28492302e4a23 (DEFAULT_TABLE table=default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef [id=163ac8a58256479c87bbc87265058cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.386238 13243 ts_tablet_manager.cc:1875] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:47.386677 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6edff8e7e6945aa926aa1486cbbc2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.386790 13246 ts_tablet_manager.cc:1888] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:47.386862 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996eb97639464e0bb976f2b1ecbbb73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.386919 13246 log.cc:1196] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/4be929f0e72e4d22a9af2fe6b3072bbe
I20240506 19:56:47.387073 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7446b0633a484cfba10f44a69dcc6969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.387236 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f424e713b96d43cdac6597c20bd5fc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.387419 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99ce389141348aa9ed9c2c56279a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.387249 13246 ts_tablet_manager.cc:1909] T 4be929f0e72e4d22a9af2fe6b3072bbe P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:47.387584 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06234a1df4264ddd9abbcaf88292795f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.387745 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5751f7370d142a2914ac75b233e8c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.387892 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1980a68925be4d3da1078c78d32da866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.388361 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96dc180faef44d598edfa11fe65e64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.389422 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbaf70315ff442e996870d9b6ddcf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.389714 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb398132c58489fb0a28492302e4a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.391592 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef5086682264414aedf3f1d50e88cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.393723 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0c085734de48a0a51f43c4e5ddf77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.394719 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b4375f10114c8085b76c7832275552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.396749 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a792bf9b774b8f8cc5491605f504ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.397406 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbcc40911034f319f489931015ca257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.401373 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d17df556f894102af1493104ac7cba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.404402 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a077a0ae1f4411c9fb2409311a9d9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.407428 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48769e0dd8864d3b85ee6a3ed1c03c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.407616 13911 raft_consensus.cc:482] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.407758 13911 raft_consensus.cc:504] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.408430 13911 leader_election.cc:288] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.408550 13911 raft_consensus.cc:482] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.408634 13911 raft_consensus.cc:504] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.409114 13911 leader_election.cc:288] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.411135 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb4ae1443ca43e98eec0339326b29d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.413419 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9b92795a7742caa2ee99ab7ca138bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.438884 13944 raft_consensus.cc:482] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.439086 13944 raft_consensus.cc:504] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.439997 13944 leader_election.cc:288] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:47.447425 12783 leader_election.cc:341] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ea25fb8e42234b82885b8fb2275d4925
I20240506 19:56:47.457278 13245 tablet_bootstrap.cc:492] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.457731 13245 ts_tablet_manager.cc:1367] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.127s user 0.023s sys 0.024s
I20240506 19:56:47.458652 13245 raft_consensus.cc:348] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.458881 13245 raft_consensus.cc:374] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.458935 13245 raft_consensus.cc:729] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.459183 13245 consensus_queue.cc:260] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.461105 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f759b560e2ab41a6a076976ccca811d1
W20240506 19:56:47.461552 12515 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.195:39651 (request call id 124) took 803 ms (client timeout 1000 ms). Trace:
W20240506 19:56:47.461685 12515 rpcz_store.cc:269] 0506 19:56:46.657918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.657980 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:47.461462 (+803482us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:47.278930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:47.279025 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:47.279030 (+ 5us) write_op.cc:435] Acquired schema lock
0506 19:56:47.279034 (+ 4us) tablet.cc:660] Decoding operations
0506 19:56:47.279106 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:47.279113 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:47.279117 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:47.279164 (+ 47us) tablet.cc:699] Row locks acquired
0506 19:56:47.279166 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:47.279190 (+ 24us) write_op.cc:270] Start()
0506 19:56:47.279212 (+ 22us) write_op.cc:276] Timestamp: P: 1715025407279173 usec, L: 0
0506 19:56:47.279215 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:47.279354 (+ 139us) log.cc:842] Serialized 533 byte log entry
0506 19:56:47.459711 (+180357us) op_driver.cc:444] REPLICATION: finished
0506 19:56:47.459956 (+ 245us) write_op.cc:301] APPLY: starting
0506 19:56:47.459980 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:47.460045 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:47.460081 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:47.460156 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:47.460208 (+ 52us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:47.460216 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:47.460325 (+ 109us) log.cc:842] Serialized 23 byte log entry
0506 19:56:47.460399 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:47.460428 (+ 29us) write_op.cc:454] Released schema lock
0506 19:56:47.460437 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:47.460503 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:47.460509 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":163,"apply.run_cpu_time_us":624,"apply.run_wall_time_us":624,"num_ops":1,"prepare.queue_time_us":150758,"prepare.run_cpu_time_us":510,"prepare.run_wall_time_us":559,"raft.queue_time_us":32,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":5,"replication_time_us":180357}]]}
I20240506 19:56:47.469828 13123 raft_consensus.cc:2454] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:47.474256 13245 ts_tablet_manager.cc:1398] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20240506 19:56:47.475581 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 624) took 1943 ms. Trace:
I20240506 19:56:47.475679 13246 rpcz_store.cc:276] 0506 19:56:45.531916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.531981 (+ 65us) service_pool.cc:225] Handling call
0506 19:56:47.180421 (+1648440us) ts_tablet_manager.cc:1120] Deleting tablet 4be929f0e72e4d22a9af2fe6b3072bbe
0506 19:56:47.180429 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.265230 (+ 84801us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.386658 (+121428us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.475556 (+ 88898us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648160}
I20240506 19:56:47.476079 13246 tablet_replica.cc:330] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:47.476296 13246 raft_consensus.cc:2229] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:47.476311 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 4be929f0e72e4d22a9af2fe6b3072bbe (table default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]) successfully deleted
I20240506 19:56:47.476454 13246 raft_consensus.cc:2258] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:47.477041 13246 ts_tablet_manager.cc:1875] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:47.478565 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d43da1d12ce647dcbcd31263932b2f54 consensus state has no leader
I20240506 19:56:47.483354 13245 tablet_bootstrap.cc:492] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.473642 14047 raft_consensus.cc:2781] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:47.384263 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe (DEFAULT_TABLE table=default.Table_51c8c97056294104afce55e84e26a550 [id=4a72c498da064f31a0c5437f8e0d1048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.384827 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 79592a4ecacb4324b2278e1b9a75496c (DEFAULT_TABLE table=default.Table_41ecc96260e940ca86684d10d102665d [id=c42c984f783c4bafaca702b9af8e42f2]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:47.385075 12921 leader_election.cc:341] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:47.492336 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 107256us
I20240506 19:56:47.493528 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.494016 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104337875517445f862d82ece3a84889" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:47.494907 12921 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:47.495134 12921 leader_election.cc:341] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.485276 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 60edeed6ae3b48e2b9f5341d7e7af01d (DEFAULT_TABLE table=default.Table_f29a4d31b66342129de3cb8f5f56f136 [id=169e562cd0814ca6b41727ca7bd648f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.498394 14034 raft_consensus.cc:504] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.498471 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb398132c58489fb0a28492302e4a23 (DEFAULT_TABLE table=default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef [id=163ac8a58256479c87bbc87265058cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.500430 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.500926 14034 leader_election.cc:288] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:47.501413 12921 leader_election.cc:341] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.501802 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a2925684b6564128a1301dfe07064f5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.502040 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_664bb720f8a04b2baf81c73ca7d3173d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.502593 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2925684b6564128a1301dfe07064f5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.502903 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_912ad4a3913f41e394764648805c3a30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.503362 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_912ad4a3913f41e394764648805c3a30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:47.502596 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_664bb720f8a04b2baf81c73ca7d3173d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.507028 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe (DEFAULT_TABLE table=default.Table_51c8c97056294104afce55e84e26a550 [id=4a72c498da064f31a0c5437f8e0d1048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.507913 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5e49e35a08cb438ebd5204ba40131503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.508455 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e49e35a08cb438ebd5204ba40131503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.509598 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_970886a5e1a34021a2a6acf328821076" } modify_external_catalogs: true
I20240506 19:56:47.509840 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_970886a5e1a34021a2a6acf328821076" } modify_external_catalogs: true
I20240506 19:56:47.510710 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 79592a4ecacb4324b2278e1b9a75496c (DEFAULT_TABLE table=default.Table_41ecc96260e940ca86684d10d102665d [id=c42c984f783c4bafaca702b9af8e42f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.516214 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet ea7e073296ce46e786362aaed96eb4dd from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:47.516336 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 9466fd3447474ca4b6f99c3810e2454e (DEFAULT_TABLE table=default.Table_1f6bae23516f4e79b0662b14c8a8ea3a [id=19bea1ba9dbf4b2787c397c6152b8c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.516394 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f29a4d31b66342129de3cb8f5f56f136" } new_table_name: "default.Table_0f648dd8cb4a4b88982f126032caf2f5" modify_external_catalogs: true
I20240506 19:56:47.518903 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 9466fd3447474ca4b6f99c3810e2454e (DEFAULT_TABLE table=default.Table_1f6bae23516f4e79b0662b14c8a8ea3a [id=19bea1ba9dbf4b2787c397c6152b8c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.485941 13245 tablet_bootstrap.cc:654] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.486043 14047 raft_consensus.cc:482] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:47.521631 14047 raft_consensus.cc:3037] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.492136 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6125a723a247a9b0bfaf55338e8a34 (DEFAULT_TABLE table=default.Table_30f6a4c5ddac420ca7fafd3366c7cd17 [id=077490f89fa147baa2713ab465729c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.494918 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 8c47a3f8d6e3440f8ab2c0b1ab49118a (DEFAULT_TABLE table=default.Table_e71ba8bf0b874298b72cae27e8607c6c [id=e450843bf078436e83034537873af619]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.488559 13094 tablet_service.cc:1467] Processing CreateTablet for tablet b8f873037284401c89dc46182aa47830 (DEFAULT_TABLE table=default.Table_9a458e9734bc4cd78dd0eb34dc07783f [id=80227a440b0948cdae9918e92486e003]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.529167 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_85795c658c664939838d720f20e0477a" } new_table_name: "default.Table_a47ab384cae5499183603602a149e649" modify_external_catalogs: true
I20240506 19:56:47.534581 13944 raft_consensus.cc:482] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 059a5c4a3f7e49ef95af4b8341702745)
I20240506 19:56:47.534778 13944 raft_consensus.cc:504] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.535470 13944 leader_election.cc:288] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:47.535753 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176c61208b1c4518820a829d9f5c0a6f consensus state has no leader
I20240506 19:56:47.539041 13245 tablet_bootstrap.cc:492] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.540705 13245 ts_tablet_manager.cc:1367] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.000s
W20240506 19:56:47.550007 12783 leader_election.cc:341] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: acd3fc62e9614cf3bde0202ee6832bb3
I20240506 19:56:47.552358 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fc5383e806314718975782b69f0243ed" } new_table_name: "default.Table_ef03c2589fde409f95b535d84b66bf29" modify_external_catalogs: true
I20240506 19:56:47.552459 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_08f77092366043de9b4041275c69dca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.553151 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08f77092366043de9b4041275c69dca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.548421 13245 raft_consensus.cc:348] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.558786 14050 raft_consensus.cc:482] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.561359 13245 raft_consensus.cc:374] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.562602 13245 raft_consensus.cc:729] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.564252 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6125a723a247a9b0bfaf55338e8a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.564705 13878 raft_consensus.cc:504] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.565400 13878 leader_election.cc:288] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.565532 13878 raft_consensus.cc:482] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.565791 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60edeed6ae3b48e2b9f5341d7e7af01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.566264 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb398132c58489fb0a28492302e4a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.566697 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.567095 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79592a4ecacb4324b2278e1b9a75496c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.562247 14050 raft_consensus.cc:504] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.565593 13245 consensus_queue.cc:260] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.567505 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9466fd3447474ca4b6f99c3810e2454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.567883 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c47a3f8d6e3440f8ab2c0b1ab49118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.568269 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f873037284401c89dc46182aa47830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.570399 13243 ts_tablet_manager.cc:1888] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:47.570525 13243 log.cc:1196] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/88dba275893f4696a5318ec21f0b2477
I20240506 19:56:47.570928 13243 ts_tablet_manager.cc:1909] T 88dba275893f4696a5318ec21f0b2477 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:47.571143 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79592a4ecacb4324b2278e1b9a75496c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.571797 14033 raft_consensus.cc:504] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.572516 14033 leader_election.cc:288] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.574692 13245 ts_tablet_manager.cc:1398] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20240506 19:56:47.575686 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 60edeed6ae3b48e2b9f5341d7e7af01d (DEFAULT_TABLE table=default.Table_f29a4d31b66342129de3cb8f5f56f136 [id=169e562cd0814ca6b41727ca7bd648f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.576150 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 with delete_type TABLET_DATA_DELETED (replaced by tablet c264c66cfec443c7a38677237ae2ba31) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.576354 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.576486 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.576615 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b89fb24d94042e1bfa772265a991680 with delete_type TABLET_DATA_DELETED (replaced by tablet dc9e8072e33347719b0f676ff6036e64) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.576738 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet f759b560e2ab41a6a076976ccca811d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.579923 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 with delete_type TABLET_DATA_DELETED (replaced by tablet c264c66cfec443c7a38677237ae2ba31) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580114 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580273 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580402 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b89fb24d94042e1bfa772265a991680 with delete_type TABLET_DATA_DELETED (replaced by tablet dc9e8072e33347719b0f676ff6036e64) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580541 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet f759b560e2ab41a6a076976ccca811d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580657 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet cd57563cc6c140c184d1725a2da6a475 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580760 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.580869 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a5a5e193f245de96cc82986e292f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.581202 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ecc91044eebb4ad6b2799b1569836c8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.581782 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecc91044eebb4ad6b2799b1569836c8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.582046 13245 tablet_bootstrap.cc:492] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.589362 14050 leader_election.cc:288] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.600570 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_da4848522d5447739671166038af13aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.601276 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da4848522d5447739671166038af13aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:47.621619 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60edeed6ae3b48e2b9f5341d7e7af01d consensus state has no leader
I20240506 19:56:47.622521 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.621601 13245 tablet_bootstrap.cc:654] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:47.624655 12921 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.625808 13944 raft_consensus.cc:482] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.625993 13944 raft_consensus.cc:504] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.626988 13944 leader_election.cc:288] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.627736 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.630021 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9466fd3447474ca4b6f99c3810e2454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.565635 13878 raft_consensus.cc:504] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:47.638455 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ee1a4a3f060407ca9c33467474e4489 consensus state has no leader
I20240506 19:56:47.638617 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60edeed6ae3b48e2b9f5341d7e7af01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.638972 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_319ec87ca4c4468aa4a97b8034323333" } modify_external_catalogs: true
I20240506 19:56:47.640875 13878 leader_election.cc:288] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.641525 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_319ec87ca4c4468aa4a97b8034323333" } modify_external_catalogs: true
I20240506 19:56:47.625579 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7440450ff02b41f78dc29fee0df8b2f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.644582 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c1ceccbbb3344105a1314370d915f2cc" } modify_external_catalogs: true
I20240506 19:56:47.644781 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c1ceccbbb3344105a1314370d915f2cc" } modify_external_catalogs: true
W20240506 19:56:47.645282 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7440450ff02b41f78dc29fee0df8b2f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:47.645713 12783 leader_election.cc:341] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a36b4d51b100486c831a41a52d81fd66
W20240506 19:56:47.654440 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1b011e94e2b47d7a51d6816576ea124 consensus state has no leader
I20240506 19:56:47.663022 12842 raft_consensus.cc:3037] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.667063 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cf03515d64254430b39e443cedd953f9" } modify_external_catalogs: true
I20240506 19:56:47.667351 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cf03515d64254430b39e443cedd953f9" } modify_external_catalogs: true
I20240506 19:56:47.669149 13243 tablet_replica.cc:330] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:47.669432 13243 raft_consensus.cc:2229] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:47.669595 13243 raft_consensus.cc:2258] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:47.670156 13243 ts_tablet_manager.cc:1875] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:47.670215 13246 ts_tablet_manager.cc:1888] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:47.670337 13246 log.cc:1196] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/db13b58d17c94e0f82052f823915d9af
I20240506 19:56:47.670644 13246 ts_tablet_manager.cc:1909] T db13b58d17c94e0f82052f823915d9af P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:47.673543 13996 raft_consensus.cc:504] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.674412 13996 leader_election.cc:288] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.674887 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 88dba275893f4696a5318ec21f0b2477 (table default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 [id=194197559e4e4469b5a3622465e90f2e]) successfully deleted
I20240506 19:56:47.678875 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 623) took 2147 ms. Trace:
I20240506 19:56:47.678982 13246 rpcz_store.cc:276] 0506 19:56:45.531727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.532311 (+ 584us) service_pool.cc:225] Handling call
0506 19:56:47.476017 (+1943706us) ts_tablet_manager.cc:1120] Deleting tablet db13b58d17c94e0f82052f823915d9af
0506 19:56:47.476030 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.564164 (+ 88134us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.670135 (+105971us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.678841 (+ 8706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943507}
I20240506 19:56:47.679472 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 630) took 2005 ms. Trace:
I20240506 19:56:47.679565 13246 rpcz_store.cc:276] 0506 19:56:45.673675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.674144 (+ 469us) service_pool.cc:225] Handling call
0506 19:56:47.679335 (+2005191us) ts_tablet_manager.cc:1120] Deleting tablet 9c19567619b64ee0a8dd637d7f69ce3e
0506 19:56:47.679346 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.679454 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2001839}
I20240506 19:56:47.679900 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 632) took 2005 ms. Trace:
I20240506 19:56:47.679975 13246 rpcz_store.cc:276] 0506 19:56:45.673981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.674216 (+ 235us) service_pool.cc:225] Handling call
0506 19:56:47.679788 (+2005572us) ts_tablet_manager.cc:1120] Deleting tablet 9944e5bfe0a741d69b8f9b6a9dad84dd
0506 19:56:47.679797 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.679885 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2001988}
I20240506 19:56:47.680284 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 633) took 2006 ms. Trace:
I20240506 19:56:47.680349 13246 rpcz_store.cc:276] 0506 19:56:45.674095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.674236 (+ 141us) service_pool.cc:225] Handling call
0506 19:56:47.680183 (+2005947us) ts_tablet_manager.cc:1120] Deleting tablet ac9ecd2e034e49e49772db7435da3c81
0506 19:56:47.680193 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.680269 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002160}
I20240506 19:56:47.680649 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 631) took 2006 ms. Trace:
I20240506 19:56:47.680720 13246 rpcz_store.cc:276] 0506 19:56:45.673859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.674193 (+ 334us) service_pool.cc:225] Handling call
0506 19:56:47.680556 (+2006363us) ts_tablet_manager.cc:1120] Deleting tablet dc206b31d3da458cb944d5f00e23d702
0506 19:56:47.680564 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.680636 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002390}
I20240506 19:56:47.680837 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet db13b58d17c94e0f82052f823915d9af (table default.Table_bb8517f5ef13428c9dcfc8c72d0cab0e [id=21f83b4858344bf9959aee15e02a7bf7]) successfully deleted
I20240506 19:56:47.681061 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 639) took 1718 ms. Trace:
I20240506 19:56:47.681131 13246 rpcz_store.cc:276] 0506 19:56:45.962307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.962529 (+ 222us) service_pool.cc:225] Handling call
0506 19:56:47.680964 (+1718435us) ts_tablet_manager.cc:1120] Deleting tablet 796beb8e0a2c41d698bd4c05796263f1
0506 19:56:47.680974 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.681045 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1718226}
W20240506 19:56:47.681257 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9944e5bfe0a741d69b8f9b6a9dad84dd already in progress: creating tablet
W20240506 19:56:47.681509 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ac9ecd2e034e49e49772db7435da3c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9ecd2e034e49e49772db7435da3c81 already in progress: creating tablet
I20240506 19:56:47.682668 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 640) took 1720 ms. Trace:
W20240506 19:56:47.683854 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet dc206b31d3da458cb944d5f00e23d702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc206b31d3da458cb944d5f00e23d702 already in progress: creating tablet
W20240506 19:56:47.684123 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 796beb8e0a2c41d698bd4c05796263f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 796beb8e0a2c41d698bd4c05796263f1 already in progress: creating tablet
I20240506 19:56:47.682996 13246 rpcz_store.cc:276] 0506 19:56:45.962462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.962777 (+ 315us) service_pool.cc:225] Handling call
0506 19:56:47.682552 (+1719775us) ts_tablet_manager.cc:1120] Deleting tablet 159ea8e0a4ee47fea93eb4b36cdd2495
0506 19:56:47.682564 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.682647 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1718392}
I20240506 19:56:47.684824 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_633a9d4ce2f94fd8b6429be9f7b1ee1d" } new_table_name: "default.Table_6ef46f8805ee4fdfa0250b1af59ef09d" modify_external_catalogs: true
I20240506 19:56:47.685586 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 641) took 1634 ms. Trace:
W20240506 19:56:47.686499 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 159ea8e0a4ee47fea93eb4b36cdd2495 already in progress: creating tablet
I20240506 19:56:47.685966 13246 rpcz_store.cc:276] 0506 19:56:46.050961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.051019 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:47.685123 (+1634104us) ts_tablet_manager.cc:1120] Deleting tablet c0c7cae21c714148a37cab389d116e91
0506 19:56:47.685135 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.685512 (+ 377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1633816}
W20240506 19:56:47.687498 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c0c7cae21c714148a37cab389d116e91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0c7cae21c714148a37cab389d116e91 already in progress: creating tablet
I20240506 19:56:47.691047 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 642) took 1626 ms. Trace:
I20240506 19:56:47.693357 13246 rpcz_store.cc:276] 0506 19:56:46.064831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.064892 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:47.689955 (+1625063us) ts_tablet_manager.cc:1120] Deleting tablet 4be4e628b9364cccb525b8e1b8070286
0506 19:56:47.690099 (+ 144us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.690892 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1624702}
I20240506 19:56:47.696734 14033 raft_consensus.cc:482] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.696969 14033 raft_consensus.cc:504] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:47.697250 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4be4e628b9364cccb525b8e1b8070286 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be4e628b9364cccb525b8e1b8070286 already in progress: creating tablet
I20240506 19:56:47.698313 14033 leader_election.cc:288] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.698879 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3e7df09f347bfa5cad4de34dbda13" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:47.699590 12921 leader_election.cc:341] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.699944 13246 tablet_replica.cc:330] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:47.703775 13246 raft_consensus.cc:2229] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:47.707685 13246 raft_consensus.cc:2258] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:47.724210 13246 ts_tablet_manager.cc:1875] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:47.731331 13245 tablet_bootstrap.cc:492] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.737548 14034 raft_consensus.cc:482] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.737821 14034 raft_consensus.cc:504] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.738683 14034 leader_election.cc:288] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:47.737932 13245 ts_tablet_manager.cc:1367] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.158s user 0.016s sys 0.003s
I20240506 19:56:47.740064 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.740211 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_28fc76ba360b4dd4b805761b13f00f04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.740792 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28fc76ba360b4dd4b805761b13f00f04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:47.741389 12921 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:47.767375 13245 raft_consensus.cc:348] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.769924 13245 raft_consensus.cc:374] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.771114 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0da0ab98ad584f7fa1dbce910bd12569" } new_table_name: "default.Table_1ed15ea0ea6148908374bd65889713a7" modify_external_catalogs: true
I20240506 19:56:47.770768 13245 raft_consensus.cc:729] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.773257 14047 raft_consensus.cc:504] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.774138 14047 leader_election.cc:288] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.772946 13245 consensus_queue.cc:260] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.777832 13245 ts_tablet_manager.cc:1398] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20240506 19:56:47.780071 13245 tablet_bootstrap.cc:492] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.797627 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f9f5211355fa4a10b23a184e7c55ae7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:47.797688 13245 tablet_bootstrap.cc:654] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:47.798285 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9f5211355fa4a10b23a184e7c55ae7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.804966 14047 raft_consensus.cc:482] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.805141 14047 raft_consensus.cc:504] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.806075 14047 leader_election.cc:288] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.806756 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5638dbb8bb784639b338a3f8d5a1b492"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.807415 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5638dbb8bb784639b338a3f8d5a1b492 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.811617 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
I20240506 19:56:47.811955 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:47.812003 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2eb5dac9d34fb09119868bf5a85803 (DEFAULT_TABLE table=default.Table_5ac61897169e4a91bbe63d32fce65440 [id=23e0f6bbcebd465ab11df2c1d9fa4e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.812479 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2eb5dac9d34fb09119868bf5a85803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.812235 12517 catalog_manager.cc:5484] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:47.812729 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:47.813056 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:47.817639 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2eb5dac9d34fb09119868bf5a85803 (DEFAULT_TABLE table=default.Table_5ac61897169e4a91bbe63d32fce65440 [id=23e0f6bbcebd465ab11df2c1d9fa4e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.818081 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2eb5dac9d34fb09119868bf5a85803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.823133 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 034b06c762cb4012ab7dc09ba7f1c41d (DEFAULT_TABLE table=default.Table_b820628699774953a9a7b655edefe41c [id=362f73af54c04bbc9de5657a30242626]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.823658 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034b06c762cb4012ab7dc09ba7f1c41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.824414 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 779534d888ab4c41bae0d7fe5fffc390 (DEFAULT_TABLE table=default.Table_0eb81c2cf9b549a28daebf834d26613c [id=49164dc63a4a4b00b435277562ee5224]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.824841 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779534d888ab4c41bae0d7fe5fffc390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.828203 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 779534d888ab4c41bae0d7fe5fffc390 (DEFAULT_TABLE table=default.Table_0eb81c2cf9b549a28daebf834d26613c [id=49164dc63a4a4b00b435277562ee5224]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.828567 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779534d888ab4c41bae0d7fe5fffc390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.838061 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_76d78f0c434c45508bfd15806c510293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.838745 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76d78f0c434c45508bfd15806c510293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.841823 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 034b06c762cb4012ab7dc09ba7f1c41d (DEFAULT_TABLE table=default.Table_b820628699774953a9a7b655edefe41c [id=362f73af54c04bbc9de5657a30242626]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.842386 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034b06c762cb4012ab7dc09ba7f1c41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.843839 13944 raft_consensus.cc:482] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.844043 13944 raft_consensus.cc:504] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.844789 13944 leader_election.cc:288] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:47.845968 12783 leader_election.cc:341] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b38453bf678748399339806d12993c88
I20240506 19:56:47.857453 13944 raft_consensus.cc:482] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.857621 13944 raft_consensus.cc:504] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:47.857933 13245 tablet_bootstrap.cc:492] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.858198 13245 ts_tablet_manager.cc:1367] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.004s
I20240506 19:56:47.858332 13944 leader_election.cc:288] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:47.859473 12783 leader_election.cc:341] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:47.865410 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.863324 13245 raft_consensus.cc:348] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.865828 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75467ef60504763a67fc10e8639716c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.865950 13245 raft_consensus.cc:374] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.866804 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.866720 13245 raft_consensus.cc:729] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.867529 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.869096 13245 consensus_queue.cc:260] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.876156 13245 ts_tablet_manager.cc:1398] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240506 19:56:47.878199 14047 raft_consensus.cc:482] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.878381 14047 raft_consensus.cc:504] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.879024 14047 leader_election.cc:288] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.880648 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbaf70315ff442e996870d9b6ddcf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.885440 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9466fd3447474ca4b6f99c3810e2454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.885705 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9b92795a7742caa2ee99ab7ca138bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.888715 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1980a68925be4d3da1078c78d32da866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.889426 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6edff8e7e6945aa926aa1486cbbc2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.892874 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb398132c58489fb0a28492302e4a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.893049 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f873037284401c89dc46182aa47830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.899015 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5294840fa14a3d88ed1fe297601d1b (DEFAULT_TABLE table=default.Table_11d89ae7590b466eacd8a986b0d180a4 [id=3cd8fd373b3e40caa343e05139cacd74]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:47.899482 13057 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.000s sys 0.011s
W20240506 19:56:47.899592 13057 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.000s sys 0.011s
I20240506 19:56:47.899793 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5294840fa14a3d88ed1fe297601d1b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:47.899818 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 561904us
I20240506 19:56:47.902566 13245 tablet_bootstrap.cc:492] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.902889 12952 tablet_service.cc:1467] Processing CreateTablet for tablet af2741336d5a4be1b5306319fc20cdc6 (DEFAULT_TABLE table=default.Table_f04d735a43fb445c9e751f02de612410 [id=88d15519330d48cbb7929cd115ce8b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.903263 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2741336d5a4be1b5306319fc20cdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.904201 14042 consensus_queue.cc:1035] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:47.904350 13092 tablet_service.cc:1467] Processing CreateTablet for tablet af2741336d5a4be1b5306319fc20cdc6 (DEFAULT_TABLE table=default.Table_f04d735a43fb445c9e751f02de612410 [id=88d15519330d48cbb7929cd115ce8b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.904781 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2741336d5a4be1b5306319fc20cdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.910046 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5294840fa14a3d88ed1fe297601d1b (DEFAULT_TABLE table=default.Table_11d89ae7590b466eacd8a986b0d180a4 [id=3cd8fd373b3e40caa343e05139cacd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.910439 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5294840fa14a3d88ed1fe297601d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.915046 13245 tablet_bootstrap.cc:654] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:47.915455 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca5dc37db5554ba4978c45ba5fd3c73e consensus state has no leader
I20240506 19:56:47.915992 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_72a13ea44ea64858911b54c9e99739fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:47.916751 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72a13ea44ea64858911b54c9e99739fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:47.919780 13245 tablet_bootstrap.cc:492] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:47.919936 13245 ts_tablet_manager.cc:1367] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240506 19:56:47.922583 13245 raft_consensus.cc:348] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.922797 13245 raft_consensus.cc:374] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:47.922855 13245 raft_consensus.cc:729] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:47.929420 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 60edeed6ae3b48e2b9f5341d7e7af01d (DEFAULT_TABLE table=default.Table_f29a4d31b66342129de3cb8f5f56f136 [id=169e562cd0814ca6b41727ca7bd648f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.932788 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 with delete_type TABLET_DATA_DELETED (replaced by tablet c264c66cfec443c7a38677237ae2ba31) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.923058 13245 consensus_queue.cc:260] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.933681 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.934077 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef5086682264414aedf3f1d50e88cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.933961 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b89fb24d94042e1bfa772265a991680 with delete_type TABLET_DATA_DELETED (replaced by tablet dc9e8072e33347719b0f676ff6036e64) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.934458 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbcc40911034f319f489931015ca257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.934661 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet f759b560e2ab41a6a076976ccca811d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.935586 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c47a3f8d6e3440f8ab2c0b1ab49118a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.935617 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 2519c03738864f6ab5dcfab7bc3fc8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.937407 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0c085734de48a0a51f43c4e5ddf77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.937850 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.938228 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06234a1df4264ddd9abbcaf88292795f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.938380 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb4ae1443ca43e98eec0339326b29d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.938542 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6125a723a247a9b0bfaf55338e8a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.939122 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79592a4ecacb4324b2278e1b9a75496c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.941459 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2eb5dac9d34fb09119868bf5a85803 (DEFAULT_TABLE table=default.Table_5ac61897169e4a91bbe63d32fce65440 [id=23e0f6bbcebd465ab11df2c1d9fa4e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.942310 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:47.942721 13245 ts_tablet_manager.cc:1398] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240506 19:56:47.943120 13245 tablet_bootstrap.cc:492] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:47.943584 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0d2620e9b044d4a5393905dd527e11 (DEFAULT_TABLE table=default.Table_9f9b64059bcb4cb0b125b84b6f163ef2 [id=a12f76991fdb4fc1923a10fb7a75e1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.944272 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0d2620e9b044d4a5393905dd527e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.946111 13245 tablet_bootstrap.cc:654] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:47.949223 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0d2620e9b044d4a5393905dd527e11 (DEFAULT_TABLE table=default.Table_9f9b64059bcb4cb0b125b84b6f163ef2 [id=a12f76991fdb4fc1923a10fb7a75e1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.950198 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0d2620e9b044d4a5393905dd527e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.950404 12961 tablet_service.cc:1467] Processing CreateTablet for tablet d00d3ca1bc504976be3f6d869390a4d3 (DEFAULT_TABLE table=default.Table_641a8c0a4cee47a18707ea343732c1d5 [id=7e017cf459b242b8bce4d98a825de531]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.952394 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48769e0dd8864d3b85ee6a3ed1c03c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.953074 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f424e713b96d43cdac6597c20bd5fc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.953275 14042 raft_consensus.cc:482] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:47.953622 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996eb97639464e0bb976f2b1ecbbb73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.953827 14042 raft_consensus.cc:504] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.953897 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 034b06c762cb4012ab7dc09ba7f1c41d (DEFAULT_TABLE table=default.Table_b820628699774953a9a7b655edefe41c [id=362f73af54c04bbc9de5657a30242626]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.954329 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7446b0633a484cfba10f44a69dcc6969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.954733 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60edeed6ae3b48e2b9f5341d7e7af01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.956805 14042 leader_election.cc:288] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:47.962630 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5294840fa14a3d88ed1fe297601d1b (DEFAULT_TABLE table=default.Table_11d89ae7590b466eacd8a986b0d180a4 [id=3cd8fd373b3e40caa343e05139cacd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.965571 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 779534d888ab4c41bae0d7fe5fffc390 (DEFAULT_TABLE table=default.Table_0eb81c2cf9b549a28daebf834d26613c [id=49164dc63a4a4b00b435277562ee5224]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.967185 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2a8e331dc24e4d06bf81c8a4b7d70156
I20240506 19:56:47.967561 12922 leader_election.cc:302] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:47.967669 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd57563cc6c140c184d1725a2da6a475" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:47.968441 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 3c882759dc0f4ba58aafaab3b855fa30 (DEFAULT_TABLE table=default.Table_351b532454104aebb7decad2864d0629 [id=224d462b1c6d4535937a0d82ff847771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.968629 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 41853001e53948a49fbb75b093dda93d (DEFAULT_TABLE table=default.Table_c09c5cd32c53422ca015e46ffcb2cfe1 [id=59d2236266164cda96142253cc0a869a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.968997 14034 raft_consensus.cc:2781] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:47.969017 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 418e02201bfc4a34a22612c2edc3e05f (DEFAULT_TABLE table=default.Table_0551b9fce765497080cf8cad54f4dc9c [id=7e16a8d752884de9a4b54a8f7a376ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.969116 14034 raft_consensus.cc:686] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:47.969369 14034 consensus_queue.cc:237] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:47.969794 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.970033 13120 raft_consensus.cc:2454] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.971012 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2417a535329341d8bb9f8218770f890d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:47.971153 13124 raft_consensus.cc:3037] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:47.971390 13095 tablet_service.cc:1467] Processing CreateTablet for tablet d00d3ca1bc504976be3f6d869390a4d3 (DEFAULT_TABLE table=default.Table_641a8c0a4cee47a18707ea343732c1d5 [id=7e017cf459b242b8bce4d98a825de531]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.971604 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 41853001e53948a49fbb75b093dda93d (DEFAULT_TABLE table=default.Table_c09c5cd32c53422ca015e46ffcb2cfe1 [id=59d2236266164cda96142253cc0a869a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.971992 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:47.973536 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104337875517445f862d82ece3a84889" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.973721 13122 raft_consensus.cc:2454] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.973995 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:47.976156 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:47.976330 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:47.976408 13123 raft_consensus.cc:2379] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:47.976528 13120 raft_consensus.cc:2379] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:47.978570 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 418e02201bfc4a34a22612c2edc3e05f (DEFAULT_TABLE table=default.Table_0551b9fce765497080cf8cad54f4dc9c [id=7e16a8d752884de9a4b54a8f7a376ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.979346 13124 raft_consensus.cc:2454] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:47.982565 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c882759dc0f4ba58aafaab3b855fa30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.982597 13243 ts_tablet_manager.cc:1888] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:47.982718 13243 log.cc:1196] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/104337875517445f862d82ece3a84889
I20240506 19:56:47.983016 13243 ts_tablet_manager.cc:1909] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:47.983337 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418e02201bfc4a34a22612c2edc3e05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.983827 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00d3ca1bc504976be3f6d869390a4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.984303 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418e02201bfc4a34a22612c2edc3e05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.984710 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00d3ca1bc504976be3f6d869390a4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.985693 13242 ts_tablet_manager.cc:1888] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:47.985805 13242 log.cc:1196] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/72a3bed0eb234e21b890242e9d23a5b4
I20240506 19:56:47.986380 13242 ts_tablet_manager.cc:1909] T 72a3bed0eb234e21b890242e9d23a5b4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:47.988374 12781 leader_election.cc:341] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2519c03738864f6ab5dcfab7bc3fc8aa
I20240506 19:56:47.988941 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5294840fa14a3d88ed1fe297601d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.989424 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2eb5dac9d34fb09119868bf5a85803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.989583 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41853001e53948a49fbb75b093dda93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.989959 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034b06c762cb4012ab7dc09ba7f1c41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.990355 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779534d888ab4c41bae0d7fe5fffc390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:47.990981 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3e7df09f347bfa5cad4de34dbda13" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.991204 13120 raft_consensus.cc:2454] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:47.988510 12781 leader_election.cc:302] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:47.968809 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad527c3f2a4b53affa928e149db1df" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:47.994418 13944 raft_consensus.cc:2726] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:47.995734 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:47.995934 13120 raft_consensus.cc:2454] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:48.004851 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0d2620e9b044d4a5393905dd527e11 (DEFAULT_TABLE table=default.Table_9f9b64059bcb4cb0b125b84b6f163ef2 [id=a12f76991fdb4fc1923a10fb7a75e1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.005546 13944 raft_consensus.cc:482] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.005686 13944 raft_consensus.cc:504] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.006314 13944 leader_election.cc:288] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.007603 12817 tablet_service.cc:1467] Processing CreateTablet for tablet d00d3ca1bc504976be3f6d869390a4d3 (DEFAULT_TABLE table=default.Table_641a8c0a4cee47a18707ea343732c1d5 [id=7e017cf459b242b8bce4d98a825de531]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:47.993902 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 15ed0c7b4569490b91c54d5bdb187e49 (DEFAULT_TABLE table=default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.011555 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 418e02201bfc4a34a22612c2edc3e05f (DEFAULT_TABLE table=default.Table_0551b9fce765497080cf8cad54f4dc9c [id=7e16a8d752884de9a4b54a8f7a376ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.012776 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ed0c7b4569490b91c54d5bdb187e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.011734 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 3c882759dc0f4ba58aafaab3b855fa30 (DEFAULT_TABLE table=default.Table_351b532454104aebb7decad2864d0629 [id=224d462b1c6d4535937a0d82ff847771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.011869 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 41853001e53948a49fbb75b093dda93d (DEFAULT_TABLE table=default.Table_c09c5cd32c53422ca015e46ffcb2cfe1 [id=59d2236266164cda96142253cc0a869a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.011936 12783 leader_election.cc:341] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:48.018867 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41853001e53948a49fbb75b093dda93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.021669 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0d2620e9b044d4a5393905dd527e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.021982 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418e02201bfc4a34a22612c2edc3e05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.022392 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00d3ca1bc504976be3f6d869390a4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.022925 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c882759dc0f4ba58aafaab3b855fa30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.030496 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:48.036577 13249 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.063s sys 0.101s
W20240506 19:56:48.037005 13249 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.698s user 0.063s sys 0.101s
I20240506 19:56:48.039714 12513 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet dc9e8072e33347719b0f676ff6036e64
I20240506 19:56:48.039909 12513 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 843) took 1098 ms. Trace:
I20240506 19:56:48.040585 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet cc22261ab2034e818834ba97c2760dad
I20240506 19:56:48.040014 12513 rpcz_store.cc:276] 0506 19:56:46.941589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.982198 (+ 40609us) service_pool.cc:225] Handling call
0506 19:56:46.982780 (+ 582us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:46.982910 (+ 130us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:46.982965 (+ 55us) catalog_manager.cc:2758] Locking tablets
0506 19:56:47.809539 (+826574us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:48.039510 (+229971us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:48.039605 (+ 95us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:48.039651 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:48.039845 (+ 194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:47.848343 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:47.848543 (+ 200us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:47.848551 (+ 8us) write_op.cc:435] Acquired schema lock
0506 19:56:47.848555 (+ 4us) tablet.cc:660] Decoding operations
0506 19:56:47.848823 (+ 268us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:47.848834 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:47.848838 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:47.849047 (+ 209us) tablet.cc:699] Row locks acquired
0506 19:56:47.849051 (+ 4us) write_op.cc:260] PREPARE: finished
0506 19:56:47.849115 (+ 64us) write_op.cc:270] Start()
0506 19:56:47.849178 (+ 63us) write_op.cc:276] Timestamp: P: 1715025407849083 usec, L: 0
0506 19:56:47.849183 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 19:56:47.863643 (+ 14460us) log.cc:842] Serialized 731 byte log entry
0506 19:56:48.038187 (+174544us) op_driver.cc:444] REPLICATION: finished
0506 19:56:48.038373 (+ 186us) write_op.cc:301] APPLY: starting
0506 19:56:48.038398 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:48.038467 (+ 69us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:48.038545 (+ 78us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:48.038628 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:48.038667 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:48.038691 (+ 24us) write_op.cc:312] APPLY: finished
0506 19:56:48.038789 (+ 98us) log.cc:842] Serialized 29 byte log entry
0506 19:56:48.038859 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:48.038886 (+ 27us) write_op.cc:454] Released schema lock
0506 19:56:48.038893 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:48.038955 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:48.038959 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":108,"apply.run_cpu_time_us":661,"apply.run_wall_time_us":662,"num_ops":2,"prepare.queue_time_us":38109,"prepare.run_cpu_time_us":1221,"prepare.run_wall_time_us":16113,"raft.queue_time_us":1146,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":188968}]]}
I20240506 19:56:48.045712 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 3c882759dc0f4ba58aafaab3b855fa30 (DEFAULT_TABLE table=default.Table_351b532454104aebb7decad2864d0629 [id=224d462b1c6d4535937a0d82ff847771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.046125 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c882759dc0f4ba58aafaab3b855fa30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.047986 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8eb9296bb1104b4cbe279eecd3683796
I20240506 19:56:48.048868 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41853001e53948a49fbb75b093dda93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.049074 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet df04fb90230d4bf4a060db993adafd14
I20240506 19:56:48.050305 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 693) took 1236 ms. Trace:
I20240506 19:56:48.050472 13243 rpcz_store.cc:276] 0506 19:56:46.814361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.814421 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:47.669051 (+854630us) ts_tablet_manager.cc:1120] Deleting tablet 104337875517445f862d82ece3a84889
0506 19:56:47.669062 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.773690 (+104628us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.982461 (+208771us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.050276 (+ 67815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":854437}
I20240506 19:56:48.051173 13243 tablet_replica.cc:330] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.051240 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet df04fb90230d4bf4a060db993adafd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.051430 13243 raft_consensus.cc:2229] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.051594 13243 raft_consensus.cc:2258] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.051864 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 104337875517445f862d82ece3a84889 (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]) successfully deleted
I20240506 19:56:48.052165 13243 ts_tablet_manager.cc:1875] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.052537 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 15ed0c7b4569490b91c54d5bdb187e49 (DEFAULT_TABLE table=default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.053220 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet cc22261ab2034e818834ba97c2760dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.053516 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.053623 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet df04fb90230d4bf4a060db993adafd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.061260 14033 raft_consensus.cc:482] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.061447 14033 raft_consensus.cc:504] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.061815 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 99f4861a01404663870ce91978c8af4f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.062090 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed with delete_type TABLET_DATA_DELETED (replaced by tablet a03f574aec984b83ab596e92a6f323fc) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.062119 14033 leader_election.cc:288] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.062260 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.062460 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.062615 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a5a5e193f245de96cc82986e292f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.062780 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet cd57563cc6c140c184d1725a2da6a475 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.062925 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.063030 12809 tablet_service.cc:1467] Processing CreateTablet for tablet af2741336d5a4be1b5306319fc20cdc6 (DEFAULT_TABLE table=default.Table_f04d735a43fb445c9e751f02de612410 [id=88d15519330d48cbb7929cd115ce8b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.063114 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 2417a535329341d8bb9f8218770f890d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:48.064198 12781 leader_election.cc:341] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ea25fb8e42234b82885b8fb2275d4925
I20240506 19:56:48.064335 12781 leader_election.cc:302] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.064771 13944 raft_consensus.cc:2726] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.064805 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet df04fb90230d4bf4a060db993adafd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.065006 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.065145 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet cc22261ab2034e818834ba97c2760dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.065433 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:48.065554 12781 leader_election.cc:341] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: acd3fc62e9614cf3bde0202ee6832bb3
I20240506 19:56:48.065670 12781 leader_election.cc:302] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.065774 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a5a5e193f245de96cc82986e292f05" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.066048 13944 raft_consensus.cc:2726] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 19:56:48.066840 13058 leader_election.cc:341] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.067077 13058 leader_election.cc:341] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.067215 13058 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.067365 13058 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.067595 13058 consensus_peers.cc:485] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:48.067718 13058 leader_election.cc:341] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.062724 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a5a5e193f245de96cc82986e292f05" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.068486 12781 leader_election.cc:341] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a36b4d51b100486c831a41a52d81fd66
I20240506 19:56:48.068578 12781 leader_election.cc:302] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.068845 13944 raft_consensus.cc:2726] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.068742 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_13e6e7d68e57433ebed02111652c43a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.069373 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13e6e7d68e57433ebed02111652c43a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.070493 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a1625cd2cd914b31b16e4b86b2aa545d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.070700 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ae36658079f748608197f569b0133127" } new_table_name: "default.Table_9aeb8fd99d7d4b9b8958a8698a439070" modify_external_catalogs: true
W20240506 19:56:48.071102 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1625cd2cd914b31b16e4b86b2aa545d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.072633 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_25774424d24c4cd68a59fd89cae9b492" } new_table_name: "default.Table_e475bad272ab485c8fa0c75f9dd69c4a" modify_external_catalogs: true
I20240506 19:56:48.073257 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 7390b6d4e5054374bed2ca2b4e4bd74f (DEFAULT_TABLE table=default.Table_8b925b5a2690446181a8ccfc20c9f959 [id=9855dc7eecda4d969b9261f88384d088]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.074741 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7390b6d4e5054374bed2ca2b4e4bd74f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.074928 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdab4110236e4f5c8690f6d963ff27f7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:48.075218 12988 raft_consensus.cc:2379] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a9c37db9d4f41cd93820a1c81f48d4d in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:56:48.075837 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ed0c7b4569490b91c54d5bdb187e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.076326 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75467ef60504763a67fc10e8639716c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.076534 12988 raft_consensus.cc:2454] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:48.076876 12516 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:48.077154 12516 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:48.078456 13088 tablet_service.cc:1467] Processing CreateTablet for tablet aab94a775b2d40568ca5c9d78db1a9cc (DEFAULT_TABLE table=default.Table_7e80a457061942da9dbca15a6b063d8e [id=b482961e445b4643a9b9e33fee0cc36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.078984 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 7390b6d4e5054374bed2ca2b4e4bd74f (DEFAULT_TABLE table=default.Table_8b925b5a2690446181a8ccfc20c9f959 [id=9855dc7eecda4d969b9261f88384d088]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.080348 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 15ed0c7b4569490b91c54d5bdb187e49 (DEFAULT_TABLE table=default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.082698 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet cc22261ab2034e818834ba97c2760dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.085810 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 473) took 5346 ms. Trace:
I20240506 19:56:48.085918 13242 rpcz_store.cc:276] 0506 19:56:42.739597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.739646 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:47.254018 (+4514372us) ts_tablet_manager.cc:1120] Deleting tablet 72a3bed0eb234e21b890242e9d23a5b4
0506 19:56:47.254051 (+ 33us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.361995 (+107944us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:47.985569 (+623574us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.085782 (+100213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4511826}
I20240506 19:56:48.086527 13242 tablet_replica.cc:330] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:48.086722 13242 raft_consensus.cc:2229] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.086916 13242 raft_consensus.cc:2258] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.087496 13242 ts_tablet_manager.cc:1875] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.077278 13249 tablet_bootstrap.cc:492] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.081596 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.090791 13249 ts_tablet_manager.cc:1367] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.757s user 0.064s sys 0.102s
I20240506 19:56:48.090958 12988 raft_consensus.cc:2454] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:48.082103 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da41ef03182445f1946584c2bf015139" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:48.082372 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd57563cc6c140c184d1725a2da6a475" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:48.091574 12986 raft_consensus.cc:3037] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.082777 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad527c3f2a4b53affa928e149db1df" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.084561 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a5a5e193f245de96cc82986e292f05" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.092238 12984 raft_consensus.cc:2454] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:48.089726 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.092633 12982 raft_consensus.cc:2454] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:48.092772 13058 leader_election.cc:341] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.093178 13057 leader_election.cc:302] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.094664 12922 leader_election.cc:302] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.094833 13057 leader_election.cc:302] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.094919 14034 raft_consensus.cc:2781] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.080101 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 7390b6d4e5054374bed2ca2b4e4bd74f (DEFAULT_TABLE table=default.Table_8b925b5a2690446181a8ccfc20c9f959 [id=9855dc7eecda4d969b9261f88384d088]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.095149 14051 raft_consensus.cc:2726] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.095302 14051 raft_consensus.cc:2781] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.095337 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7390b6d4e5054374bed2ca2b4e4bd74f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.095371 14051 raft_consensus.cc:482] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.095470 14051 raft_consensus.cc:3037] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.095412 14047 consensus_queue.cc:1035] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:48.079903 12955 tablet_service.cc:1467] Processing CreateTablet for tablet aab94a775b2d40568ca5c9d78db1a9cc (DEFAULT_TABLE table=default.Table_7e80a457061942da9dbca15a6b063d8e [id=b482961e445b4643a9b9e33fee0cc36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.096472 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab94a775b2d40568ca5c9d78db1a9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.096683 13057 leader_election.cc:302] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.096913 13878 raft_consensus.cc:2781] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.096987 13878 raft_consensus.cc:482] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.097048 13878 raft_consensus.cc:3037] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:48.097059 13057 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: Tablet not RUNNING: STOPPED
I20240506 19:56:48.097146 13057 leader_election.cc:302] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.097389 14047 raft_consensus.cc:2726] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.099282 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 72a3bed0eb234e21b890242e9d23a5b4 (table default.Table_fdae8810fa204843b7e162d421c80856 [id=6c23705f0cdc4161b2a33e4fb164e742]) successfully deleted
W20240506 19:56:48.099931 13057 leader_election.cc:341] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.100205 13057 leader_election.cc:302] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.100466 14047 raft_consensus.cc:2726] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.100714 13057 leader_election.cc:302] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:48.101074 14047 raft_consensus.cc:2781] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.101161 14047 raft_consensus.cc:482] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.101238 14047 raft_consensus.cc:3037] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.101696 13057 leader_election.cc:302] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:48.102340 12516 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 3 got 0
I20240506 19:56:48.095081 14034 raft_consensus.cc:482] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.104457 14034 raft_consensus.cc:3037] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.104296 13249 raft_consensus.cc:348] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.104522 13249 raft_consensus.cc:374] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.104565 13249 raft_consensus.cc:729] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:48.105086 13058 leader_election.cc:341] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.107110 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ceed94ec290a471ca0114a9727776f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.107287 13058 consensus_peers.cc:485] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:48.107888 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceed94ec290a471ca0114a9727776f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.108480 13246 ts_tablet_manager.cc:1888] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.108609 13246 log.cc:1196] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/e17197adc6ca4593ac3fc494548a6e67
I20240506 19:56:48.108912 13246 ts_tablet_manager.cc:1909] T e17197adc6ca4593ac3fc494548a6e67 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.109383 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab94a775b2d40568ca5c9d78db1a9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.109707 12827 tablet_service.cc:1467] Processing CreateTablet for tablet e1d7a850a3e644c3a38ab224efcc8af5 (DEFAULT_TABLE table=default.Table_fac42373406c413f9eb5907032a077f9 [id=379a3d3f4e7f4ef0b38991134b9006f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.109899 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7390b6d4e5054374bed2ca2b4e4bd74f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.109022 13249 consensus_queue.cc:260] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.111173 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_54bc24e04c80487d9a93fd01ad679d73" } new_table_name: "default.Table_be62a1b5f7694bb1b6e32c3e5a843906" modify_external_catalogs: true
I20240506 19:56:48.112381 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_36502249055f419692c8fe55e7f7d2ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.112851 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36502249055f419692c8fe55e7f7d2ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.113382 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.113479 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 44999.4us
W20240506 19:56:48.114022 12781 leader_election.cc:341] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b38453bf678748399339806d12993c88
I20240506 19:56:48.114014 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_538bc8e0e1604e09bed86b488cf7ada0" } new_table_name: "default.Table_898e8b5ca87647bf9715e1304ed7f90b" modify_external_catalogs: true
I20240506 19:56:48.114132 12781 leader_election.cc:302] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:48.114862 12921 leader_election.cc:341] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.115144 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c35eaf4f2426412381b537c1bcb048f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.115636 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c35eaf4f2426412381b537c1bcb048f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.117026 13249 ts_tablet_manager.cc:1398] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240506 19:56:48.117345 13249 tablet_bootstrap.cc:492] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.118443 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a5a5e193f245de96cc82986e292f05" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.119593 12965 tablet_service.cc:1467] Processing CreateTablet for tablet b78c7c1cafac46c4b9e2750c41d24620 (DEFAULT_TABLE table=default.Table_b5789735f41642769315e305be4cc6ca [id=c4798c42209a4c5eabce8b5053715386]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.119721 12507 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 58c88f80f5b34672b7c3a4b604469bf1 (table default.Table_9197190885234701b5933420d98f8ca8 [id=d25fcde9ffaa4b64a0322b921aacc8d4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 58c88f80f5b34672b7c3a4b604469bf1 consensus state has no leader
I20240506 19:56:48.120577 13091 tablet_service.cc:1467] Processing CreateTablet for tablet e1d7a850a3e644c3a38ab224efcc8af5 (DEFAULT_TABLE table=default.Table_fac42373406c413f9eb5907032a077f9 [id=379a3d3f4e7f4ef0b38991134b9006f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.121094 13103 tablet_service.cc:1467] Processing CreateTablet for tablet ce6483470c864697be6d9590ec95edfc (DEFAULT_TABLE table=default.Table_857904c3c3954ef5a113de49478bd27f [id=eb8c03717dc1454e8a944e84f309c31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.121721 12952 tablet_service.cc:1467] Processing CreateTablet for tablet ce6483470c864697be6d9590ec95edfc (DEFAULT_TABLE table=default.Table_857904c3c3954ef5a113de49478bd27f [id=eb8c03717dc1454e8a944e84f309c31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.122190 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ed0c7b4569490b91c54d5bdb187e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.123370 13944 raft_consensus.cc:2726] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.124105 13245 tablet_bootstrap.cc:492] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.124228 13245 ts_tablet_manager.cc:1367] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.181s user 0.006s sys 0.019s
I20240506 19:56:48.124933 13245 raft_consensus.cc:348] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.125124 13245 raft_consensus.cc:374] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.125222 13245 raft_consensus.cc:729] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:48.129259 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb4945e5757c401cacb39037459e2b57 consensus state has no leader
W20240506 19:56:48.129654 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e63baf59dc8244daad831b2987a51cfb consensus state has no leader
I20240506 19:56:48.133473 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6483470c864697be6d9590ec95edfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.133983 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d7a850a3e644c3a38ab224efcc8af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.125456 13245 consensus_queue.cc:260] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.137063 14072 raft_consensus.cc:2781] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.137141 14072 raft_consensus.cc:482] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.137207 14072 raft_consensus.cc:3037] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.138077 13245 ts_tablet_manager.cc:1398] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240506 19:56:48.138365 13245 tablet_bootstrap.cc:492] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.139866 13245 tablet_bootstrap.cc:654] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.134115 12813 tablet_service.cc:1467] Processing CreateTablet for tablet aab94a775b2d40568ca5c9d78db1a9cc (DEFAULT_TABLE table=default.Table_7e80a457061942da9dbca15a6b063d8e [id=b482961e445b4643a9b9e33fee0cc36b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.141603 13058 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.144296 12953 tablet_service.cc:1467] Processing CreateTablet for tablet d4c8fb0fb09c4d32b8335fc6bd0827c7 (DEFAULT_TABLE table=default.Table_ff86f5a848874ce7960fbcc9c79457fd [id=469bb2f1eb87405aab9296077b9130ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.144745 12962 tablet_service.cc:1467] Processing CreateTablet for tablet ca5dc37db5554ba4978c45ba5fd3c73e (DEFAULT_TABLE table=default.Table_1ed15ea0ea6148908374bd65889713a7 [id=757c6015872c4aa58e3fbd44122e2ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.147102 13098 tablet_service.cc:1467] Processing CreateTablet for tablet b78c7c1cafac46c4b9e2750c41d24620 (DEFAULT_TABLE table=default.Table_b5789735f41642769315e305be4cc6ca [id=c4798c42209a4c5eabce8b5053715386]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.147465 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78c7c1cafac46c4b9e2750c41d24620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.147492 13249 tablet_bootstrap.cc:654] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.149591 13094 tablet_service.cc:1467] Processing CreateTablet for tablet d4c8fb0fb09c4d32b8335fc6bd0827c7 (DEFAULT_TABLE table=default.Table_ff86f5a848874ce7960fbcc9c79457fd [id=469bb2f1eb87405aab9296077b9130ac]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.149243 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 54598.7us
I20240506 19:56:48.149914 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c8fb0fb09c4d32b8335fc6bd0827c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.150142 13245 tablet_bootstrap.cc:492] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.150087 12922 leader_election.cc:302] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.150254 13245 ts_tablet_manager.cc:1367] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
W20240506 19:56:48.150285 12922 raft_consensus.cc:2670] T 104337875517445f862d82ece3a84889 P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:48.150940 12922 leader_election.cc:302] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.151149 14033 raft_consensus.cc:2726] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.151320 12922 leader_election.cc:302] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:48.151465 12922 raft_consensus.cc:2670] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:48.151652 13245 raft_consensus.cc:348] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.151754 12922 leader_election.cc:302] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.151832 13245 raft_consensus.cc:374] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.151885 13245 raft_consensus.cc:729] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.152066 13245 consensus_queue.cc:260] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.153002 14033 raft_consensus.cc:2781] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.153126 14033 raft_consensus.cc:686] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:48.153026 14068 tablet.cc:1722] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:48.153399 14033 consensus_queue.cc:237] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.153543 13101 tablet_service.cc:1467] Processing CreateTablet for tablet ca5dc37db5554ba4978c45ba5fd3c73e (DEFAULT_TABLE table=default.Table_1ed15ea0ea6148908374bd65889713a7 [id=757c6015872c4aa58e3fbd44122e2ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.153859 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5dc37db5554ba4978c45ba5fd3c73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.162895 13245 ts_tablet_manager.cc:1398] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240506 19:56:48.163219 13245 tablet_bootstrap.cc:492] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.164822 13245 tablet_bootstrap.cc:654] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.167001 12922 leader_election.cc:302] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:48.170753 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 56730.4us
W20240506 19:56:48.171030 12781 leader_election.cc:341] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:48.171141 12781 leader_election.cc:302] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.171370 13944 raft_consensus.cc:2726] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:48.171907 12921 consensus_peers.cc:485] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:48.172313 14033 raft_consensus.cc:2781] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.172398 14033 raft_consensus.cc:482] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.172472 14033 raft_consensus.cc:3037] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.177659 13092 tablet_service.cc:1514] Processing DeleteTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.177940 12817 tablet_service.cc:1467] Processing CreateTablet for tablet ce6483470c864697be6d9590ec95edfc (DEFAULT_TABLE table=default.Table_857904c3c3954ef5a113de49478bd27f [id=eb8c03717dc1454e8a944e84f309c31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.177529 14061 tablet.cc:1722] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:48.179419 12922 leader_election.cc:302] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.180471 13878 raft_consensus.cc:504] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.180606 12986 raft_consensus.cc:2454] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:48.181281 13878 leader_election.cc:288] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.183840 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad527c3f2a4b53affa928e149db1df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.184794 14034 raft_consensus.cc:504] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.186172 14034 leader_election.cc:288] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.187252 12961 tablet_service.cc:1467] Processing CreateTablet for tablet e1d7a850a3e644c3a38ab224efcc8af5 (DEFAULT_TABLE table=default.Table_fac42373406c413f9eb5907032a077f9 [id=379a3d3f4e7f4ef0b38991134b9006f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.187507 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2741336d5a4be1b5306319fc20cdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.187819 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab94a775b2d40568ca5c9d78db1a9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.188089 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d7a850a3e644c3a38ab224efcc8af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.188596 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6483470c864697be6d9590ec95edfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.188792 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78c7c1cafac46c4b9e2750c41d24620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.189000 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5dc37db5554ba4978c45ba5fd3c73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.189399 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c8fb0fb09c4d32b8335fc6bd0827c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.190652 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d7a850a3e644c3a38ab224efcc8af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.192044 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6483470c864697be6d9590ec95edfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.194219 12822 tablet_service.cc:1467] Processing CreateTablet for tablet b78c7c1cafac46c4b9e2750c41d24620 (DEFAULT_TABLE table=default.Table_b5789735f41642769315e305be4cc6ca [id=c4798c42209a4c5eabce8b5053715386]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.194986 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.227063 13058 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.227854 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.228468 12921 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.228871 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0027f2a42b1a494685c82f0862604186" } new_table_name: "default.Table_f11c6836fece4a3fb0a20a2ad29761a3" modify_external_catalogs: true
I20240506 19:56:48.232460 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_74493d529d484630900cd65b7b1777da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.233040 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74493d529d484630900cd65b7b1777da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.234537 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_29547e0639d84d1d803dd3f2a9fed320"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.235038 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29547e0639d84d1d803dd3f2a9fed320 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.236202 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1258c1a3736349b0973af069d7d58b02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.236629 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1258c1a3736349b0973af069d7d58b02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.245867 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:48.246276 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]). Expected version 3 got 0
I20240506 19:56:48.250591 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78c7c1cafac46c4b9e2750c41d24620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.250087 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9ecd2e034e49e49772db7435da3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.253662 14080 raft_consensus.cc:2781] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.254154 13878 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.254298 13878 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.254981 13878 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:48.255251 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffbb87878d06417cbabf3c218728dbdc consensus state has no leader
I20240506 19:56:48.256485 13245 tablet_bootstrap.cc:492] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.256632 13245 ts_tablet_manager.cc:1367] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.093s user 0.001s sys 0.002s
I20240506 19:56:48.257385 13245 raft_consensus.cc:348] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.257601 13245 raft_consensus.cc:374] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.257669 13245 raft_consensus.cc:729] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.257860 13245 consensus_queue.cc:260] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.258369 13245 ts_tablet_manager.cc:1398] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:48.259791 13245 tablet_bootstrap.cc:492] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.261076 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a5d3e2fa72f2440eae3408f5b092bd7c" } modify_external_catalogs: true
I20240506 19:56:48.261266 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a5d3e2fa72f2440eae3408f5b092bd7c" } modify_external_catalogs: true
I20240506 19:56:48.262027 13245 tablet_bootstrap.cc:654] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.263072 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3dea609720914447924d3591bfed24f6" } modify_external_catalogs: true
I20240506 19:56:48.263257 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3dea609720914447924d3591bfed24f6" } modify_external_catalogs: true
I20240506 19:56:48.263586 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7ae101bd10f242f28c19bd685449fce8" } modify_external_catalogs: true
I20240506 19:56:48.263706 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7ae101bd10f242f28c19bd685449fce8" } modify_external_catalogs: true
I20240506 19:56:48.264879 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_31c22580ea6a414da4a17078ed4f88ef" } modify_external_catalogs: true
I20240506 19:56:48.265055 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_31c22580ea6a414da4a17078ed4f88ef" } modify_external_catalogs: true
I20240506 19:56:48.265969 12823 tablet_service.cc:1467] Processing CreateTablet for tablet d4c8fb0fb09c4d32b8335fc6bd0827c7 (DEFAULT_TABLE table=default.Table_ff86f5a848874ce7960fbcc9c79457fd [id=469bb2f1eb87405aab9296077b9130ac]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.269389 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1a792bf9b774b8f8cc5491605f504ac consensus state has no leader
I20240506 19:56:48.270668 14080 raft_consensus.cc:482] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.271136 14080 raft_consensus.cc:3037] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:48.271325 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f55586b153b44c31a298a8d46bd8c532 consensus state has no leader
I20240506 19:56:48.271692 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.274560 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 3 got 0
W20240506 19:56:48.274808 13058 leader_election.cc:341] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.275152 13245 tablet_bootstrap.cc:492] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.275278 13245 ts_tablet_manager.cc:1367] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20240506 19:56:48.275681 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9c5d2e15a9e64b618f68377dce21b541"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.276016 13245 raft_consensus.cc:348] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.276185 13245 raft_consensus.cc:374] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.276243 13245 raft_consensus.cc:729] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:48.276232 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c5d2e15a9e64b618f68377dce21b541 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.276471 13245 consensus_queue.cc:260] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.276980 13245 ts_tablet_manager.cc:1398] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:48.277256 13245 tablet_bootstrap.cc:492] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.277246 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_08f77092366043de9b4041275c69dca2" } modify_external_catalogs: true
I20240506 19:56:48.277431 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_08f77092366043de9b4041275c69dca2" } modify_external_catalogs: true
I20240506 19:56:48.277263 12811 tablet_service.cc:1467] Processing CreateTablet for tablet ca5dc37db5554ba4978c45ba5fd3c73e (DEFAULT_TABLE table=default.Table_1ed15ea0ea6148908374bd65889713a7 [id=757c6015872c4aa58e3fbd44122e2ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.279421 13245 tablet_bootstrap.cc:654] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.280689 14033 raft_consensus.cc:504] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.281320 14033 leader_election.cc:288] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.281481 14033 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.281581 14033 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.282724 13243 ts_tablet_manager.cc:1888] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.282712 14047 raft_consensus.cc:504] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.283326 14047 leader_election.cc:288] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.283679 14051 raft_consensus.cc:504] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.284197 14051 leader_election.cc:288] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.284348 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 643) took 2217 ms. Trace:
I20240506 19:56:48.284449 13246 rpcz_store.cc:276] 0506 19:56:46.067087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.067144 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:47.699110 (+1631966us) ts_tablet_manager.cc:1120] Deleting tablet e17197adc6ca4593ac3fc494548a6e67
0506 19:56:47.699272 (+ 162us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:47.985166 (+285894us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.108324 (+123158us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.284327 (+176003us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631381}
I20240506 19:56:48.284772 13246 tablet_replica.cc:330] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.284940 13246 raft_consensus.cc:2229] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.285058 13246 raft_consensus.cc:2258] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.285645 13246 ts_tablet_manager.cc:1875] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.286792 14072 raft_consensus.cc:504] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.287320 14072 leader_election.cc:288] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.287475 13120 raft_consensus.cc:2379] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:48.291157 13245 tablet_bootstrap.cc:492] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.291281 13245 ts_tablet_manager.cc:1367] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20240506 19:56:48.292269 13245 raft_consensus.cc:348] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.292454 13245 raft_consensus.cc:374] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.292503 13245 raft_consensus.cc:729] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.292686 13245 consensus_queue.cc:260] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.293252 13245 ts_tablet_manager.cc:1398] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:48.293536 13245 tablet_bootstrap.cc:492] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.297611 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet e17197adc6ca4593ac3fc494548a6e67 (table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]) successfully deleted
I20240506 19:56:48.298923 13245 tablet_bootstrap.cc:654] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.301213 14033 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.303176 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:48.303524 13243 log.cc:1196] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/da41ef03182445f1946584c2bf015139
W20240506 19:56:48.306152 13058 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.307066 13244 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.046s sys 0.126s
I20240506 19:56:48.309928 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75467ef60504763a67fc10e8639716c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.314054 13058 leader_election.cc:341] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.318686 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_112c674a2f6a46cb8e60669eb015488f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.319487 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_112c674a2f6a46cb8e60669eb015488f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.322353 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 3 got 2
W20240506 19:56:48.322871 13244 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.969s user 0.046s sys 0.126s
I20240506 19:56:48.324172 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a5a5e193f245de96cc82986e292f05" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.325219 13058 leader_election.cc:341] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.327185 13244 tablet_bootstrap.cc:492] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:48.327857 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bd3130bc41084bfaabaaecc0985e378d
I20240506 19:56:48.329241 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3130bc41084bfaabaaecc0985e378d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.329329 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3130bc41084bfaabaaecc0985e378d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.329839 13244 ts_tablet_manager.cc:1367] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.992s user 0.047s sys 0.127s
I20240506 19:56:48.330114 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3130bc41084bfaabaaecc0985e378d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.330523 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5dc37db5554ba4978c45ba5fd3c73e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:48.330686 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 65624.3us
I20240506 19:56:48.331310 12516 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885] alter complete (version 3)
I20240506 19:56:48.333070 14090 tablet.cc:1722] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:48.336237 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.337114 12921 leader_election.cc:341] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.337777 13243 ts_tablet_manager.cc:1909] T da41ef03182445f1946584c2bf015139 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.337803 14061 tablet.cc:1722] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:48.338596 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 694) took 1490 ms. Trace:
I20240506 19:56:48.338634 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 378759ca310f453fba45d48879e2dd1a with delete_type TABLET_DATA_DELETED (replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.338690 13243 rpcz_store.cc:276] 0506 19:56:46.848549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.848612 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:48.051121 (+1202509us) ts_tablet_manager.cc:1120] Deleting tablet da41ef03182445f1946584c2bf015139
0506 19:56:48.051131 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.190623 (+139492us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.282623 (+ 92000us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.338576 (+ 55953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202207}
I20240506 19:56:48.339504 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet da41ef03182445f1946584c2bf015139 (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
I20240506 19:56:48.345124 13246 ts_tablet_manager.cc:1888] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.345391 13246 log.cc:1196] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/517c09e3bf7046ccba6ed25b08e3460e
I20240506 19:56:48.346479 13246 ts_tablet_manager.cc:1909] T 517c09e3bf7046ccba6ed25b08e3460e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.348498 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 644) took 2280 ms. Trace:
I20240506 19:56:48.348645 13246 rpcz_store.cc:276] 0506 19:56:46.067663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.067739 (+ 76us) service_pool.cc:225] Handling call
0506 19:56:48.284729 (+2216990us) ts_tablet_manager.cc:1120] Deleting tablet 517c09e3bf7046ccba6ed25b08e3460e
0506 19:56:48.284738 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.288622 (+ 3884us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.344903 (+ 56281us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.348437 (+ 3534us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216828}
I20240506 19:56:48.348958 12804 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 131) took 1111 ms. Trace:
I20240506 19:56:48.349151 12804 rpcz_store.cc:276] 0506 19:56:47.224591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.226073 (+ 1482us) service_pool.cc:225] Handling call
0506 19:56:48.336280 (+1110207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:48.349392 13246 tablet_replica.cc:330] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.349738 13246 raft_consensus.cc:2229] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.349953 13246 raft_consensus.cc:2258] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.350782 13246 ts_tablet_manager.cc:1875] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.343626 14080 raft_consensus.cc:504] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.355502 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 517c09e3bf7046ccba6ed25b08e3460e (table default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 [id=4b18d70e1c8847de81b7742f55f56260]) successfully deleted
I20240506 19:56:48.356398 14080 leader_election.cc:288] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.359560 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 574dbbe8a9a94cd7944b652a68b77403
I20240506 19:56:48.361213 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 574dbbe8a9a94cd7944b652a68b77403 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.339035 13243 tablet_replica.cc:330] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.363004 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:48.363391 13243 raft_consensus.cc:2229] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:48.336254 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3e7df09f347bfa5cad4de34dbda13" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.364658 12921 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.365067 12921 leader_election.cc:341] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.366815 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7763959136c44674888916321bd6da24
I20240506 19:56:48.366621 13243 pending_rounds.cc:62] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:48.366906 13243 pending_rounds.cc:68] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744071045210112 op_type: NO_OP noop_request { }
I20240506 19:56:48.367636 13244 raft_consensus.cc:348] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.368013 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 7763959136c44674888916321bd6da24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.368489 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c8fb0fb09c4d32b8335fc6bd0827c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.369941 13243 raft_consensus.cc:2866] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:48.369959 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 574dbbe8a9a94cd7944b652a68b77403 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.369975 13244 raft_consensus.cc:374] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.370069 13243 raft_consensus.cc:2258] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.370071 13244 raft_consensus.cc:729] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.370178 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 7763959136c44674888916321bd6da24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.370316 13244 consensus_queue.cc:260] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.370836 13244 ts_tablet_manager.cc:1398] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20240506 19:56:48.371388 13243 ts_tablet_manager.cc:1875] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.377070 14072 raft_consensus.cc:482] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.377229 14072 raft_consensus.cc:504] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.377251 13243 ts_tablet_manager.cc:1888] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:48.377923 14072 leader_election.cc:288] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.378057 13243 log.cc:1196] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a8b3ed0b56aa478abb44347afca8fcf6
I20240506 19:56:48.381898 13243 ts_tablet_manager.cc:1909] T a8b3ed0b56aa478abb44347afca8fcf6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.383754 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 7763959136c44674888916321bd6da24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.384025 13245 tablet_bootstrap.cc:492] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.384167 13245 ts_tablet_manager.cc:1367] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.091s user 0.028s sys 0.028s
I20240506 19:56:48.384680 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 695) took 1482 ms. Trace:
I20240506 19:56:48.384773 13243 rpcz_store.cc:276] 0506 19:56:46.902193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.902264 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:48.338983 (+1436719us) ts_tablet_manager.cc:1120] Deleting tablet a8b3ed0b56aa478abb44347afca8fcf6
0506 19:56:48.338994 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.374402 (+ 35408us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.377170 (+ 2768us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.384660 (+ 7490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436464}
I20240506 19:56:48.385107 13243 tablet_replica.cc:330] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.385285 13243 raft_consensus.cc:2229] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:48.385630 13243 pending_rounds.cc:62] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:48.385695 13243 pending_rounds.cc:68] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744067757916160 op_type: NO_OP noop_request { }
I20240506 19:56:48.385841 13243 raft_consensus.cc:2866] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:48.385942 13243 raft_consensus.cc:2258] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.386495 13246 ts_tablet_manager.cc:1888] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.386617 13246 log.cc:1196] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:48.386920 13246 ts_tablet_manager.cc:1909] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.387300 13245 raft_consensus.cc:348] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.387493 13245 raft_consensus.cc:374] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.387553 13245 raft_consensus.cc:729] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.387768 13245 consensus_queue.cc:260] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.388478 13245 ts_tablet_manager.cc:1398] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240506 19:56:48.389117 13243 ts_tablet_manager.cc:1875] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.390691 13245 tablet_bootstrap.cc:492] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.390712 12522 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 3cdd4cb58d4d4f00b5d100350d22d26c
I20240506 19:56:48.391101 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03df4632adfa4b0ab999aeaa5bb4e1f4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.392827 13245 tablet_bootstrap.cc:654] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.393939 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 645) took 2325 ms. Trace:
I20240506 19:56:48.394016 13246 rpcz_store.cc:276] 0506 19:56:46.068194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.068244 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:48.349272 (+2281028us) ts_tablet_manager.cc:1120] Deleting tablet 6d04a1c5ee7b4d529fa2f4da4945ff09
0506 19:56:48.349285 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.374776 (+ 25491us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.386378 (+ 11602us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.393916 (+ 7538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280843}
I20240506 19:56:48.394762 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfe68f19a2874be5bdd3c0321ba9872d" } new_table_name: "default.Table_c364eee867924659ae35358d322b91ae" modify_external_catalogs: true
I20240506 19:56:48.394753 13246 tablet_replica.cc:330] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.395097 14072 raft_consensus.cc:482] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.395247 14072 raft_consensus.cc:504] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.395840 14072 leader_election.cc:288] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.399086 13243 ts_tablet_manager.cc:1888] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:48.399036 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9a6d6630a6d643548740709cfeb1deb0" } new_table_name: "default.Table_516b51665d1d453491f1c9caf37dda83" modify_external_catalogs: true
I20240506 19:56:48.399195 13243 log.cc:1196] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:48.399551 13243 ts_tablet_manager.cc:1909] T 7d23f0a5ff6d430282256e9bddc186b9 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.401587 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]). Expected version 3 got 2
I20240506 19:56:48.402614 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_964a0d50cb2d48fa988072257bec9ec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.403148 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_964a0d50cb2d48fa988072257bec9ec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.403270 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 696) took 1445 ms. Trace:
I20240506 19:56:48.403344 13243 rpcz_store.cc:276] 0506 19:56:46.958311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.958367 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:48.385059 (+1426692us) ts_tablet_manager.cc:1120] Deleting tablet 7d23f0a5ff6d430282256e9bddc186b9
0506 19:56:48.385071 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.395804 (+ 10733us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.398980 (+ 3176us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.403247 (+ 4267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426489}
I20240506 19:56:48.403733 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 698) took 1257 ms. Trace:
I20240506 19:56:48.403821 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 7d23f0a5ff6d430282256e9bddc186b9 (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:48.403802 13243 rpcz_store.cc:276] 0506 19:56:47.146018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.146081 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:48.403604 (+1257523us) ts_tablet_manager.cc:1120] Deleting tablet 01c32eba2031446ab419cac64068068e
0506 19:56:48.403614 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.403710 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257316}
I20240506 19:56:48.405753 13243 tablet_replica.cc:330] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.405943 13243 raft_consensus.cc:2229] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:48.406018 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 01c32eba2031446ab419cac64068068e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01c32eba2031446ab419cac64068068e already in progress: creating tablet
I20240506 19:56:48.406121 13243 raft_consensus.cc:2258] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.408000 13243 ts_tablet_manager.cc:1875] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:48.409744 13058 leader_election.cc:341] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.411022 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bdd8b0bb2f654920a49117008a662af1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.411682 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdd8b0bb2f654920a49117008a662af1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.417517 13243 ts_tablet_manager.cc:1888] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.417662 13243 log.cc:1196] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/fdab4110236e4f5c8690f6d963ff27f7
I20240506 19:56:48.418021 13243 ts_tablet_manager.cc:1909] T fdab4110236e4f5c8690f6d963ff27f7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.418927 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 699) took 1268 ms. Trace:
I20240506 19:56:48.419054 13243 rpcz_store.cc:276] 0506 19:56:47.150254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.150338 (+ 84us) service_pool.cc:225] Handling call
0506 19:56:48.405708 (+1255370us) ts_tablet_manager.cc:1120] Deleting tablet fdab4110236e4f5c8690f6d963ff27f7
0506 19:56:48.405718 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.414516 (+ 8798us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.417410 (+ 2894us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.418902 (+ 1492us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255071}
I20240506 19:56:48.419445 13243 tablet_replica.cc:330] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.419626 13243 raft_consensus.cc:2229] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.419610 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet fdab4110236e4f5c8690f6d963ff27f7 (table default.Table_b4ce4fd613b049df9285a95fe23671c1 [id=89681fb9b58c4f429a4f94873f3827ff]) successfully deleted
I20240506 19:56:48.419775 13243 raft_consensus.cc:2258] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.420347 13243 ts_tablet_manager.cc:1875] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.421207 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.421373 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fb091762144f6591152680a6acd29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.421829 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7397db1e74b4488a3f9a70136bccb with delete_type TABLET_DATA_DELETED (replaced by tablet 65315d8ac5ff408bae8593681335e5b1) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.421828 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 (table default.Table_bece2f866f8449ad8ef2f48007dc14ef [id=4021900d99ce4302b2de9b33a687e94f]) successfully deleted
I20240506 19:56:48.422375 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet b5a443c0d35741b9966b32a7808a97e2 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:48.276870 13249 tablet_bootstrap.cc:492] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.423779 13249 ts_tablet_manager.cc:1367] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.307s user 0.002s sys 0.010s
I20240506 19:56:48.424628 13249 raft_consensus.cc:348] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.424870 13249 raft_consensus.cc:374] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.424968 13249 raft_consensus.cc:729] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.425186 13249 consensus_queue.cc:260] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.426189 13249 ts_tablet_manager.cc:1398] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:48.426600 13249 tablet_bootstrap.cc:492] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.427853 13243 ts_tablet_manager.cc:1888] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.427981 13243 log.cc:1196] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:48.428309 13243 ts_tablet_manager.cc:1909] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:48.429661 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 260454us
I20240506 19:56:48.430651 12922 leader_election.cc:302] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.430960 14075 raft_consensus.cc:3037] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.434711 14075 raft_consensus.cc:2726] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.434903 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 700) took 1282 ms. Trace:
I20240506 19:56:48.434993 13243 rpcz_store.cc:276] 0506 19:56:47.152916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.153074 (+ 158us) service_pool.cc:225] Handling call
0506 19:56:48.419396 (+1266322us) ts_tablet_manager.cc:1120] Deleting tablet 2b5a46b91cb742a994496648fbf87aa4
0506 19:56:48.419407 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.424607 (+ 5200us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.427766 (+ 3159us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.434885 (+ 7119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266104}
I20240506 19:56:48.435549 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2b5a46b91cb742a994496648fbf87aa4 (table default.Table_3a6afcad38024709a1f104fa5bbadcff [id=2249bb6d1f824bacbd5b5b1c86e58c48]) successfully deleted
I20240506 19:56:48.437541 13243 tablet_replica.cc:330] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.437726 13243 raft_consensus.cc:2229] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.438252 13243 raft_consensus.cc:2258] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.438772 13243 ts_tablet_manager.cc:1875] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.438741 13057 leader_election.cc:302] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.439019 13244 tablet_bootstrap.cc:492] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.439496 14072 raft_consensus.cc:2781] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.439622 14072 raft_consensus.cc:686] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:48.439863 14072 consensus_queue.cc:237] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.440065 14090 tablet.cc:1722] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:48.442876 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6a2c98fd374b8cbc5a65069bd4cb5b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.444224 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 574dbbe8a9a94cd7944b652a68b77403 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.445762 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.445958 12982 raft_consensus.cc:2454] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:48.446199 13243 ts_tablet_manager.cc:1888] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.446301 13243 log.cc:1196] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:48.446592 13243 ts_tablet_manager.cc:1909] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.446681 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a5a5e193f245de96cc82986e292f05" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
W20240506 19:56:48.446801 13058 leader_election.cc:341] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.447039 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 707) took 1217 ms. Trace:
I20240506 19:56:48.447129 13243 rpcz_store.cc:276] 0506 19:56:47.229555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.230713 (+ 1158us) service_pool.cc:225] Handling call
0506 19:56:48.435280 (+1204567us) ts_tablet_manager.cc:1120] Deleting tablet 9e3d13322b4a4a2db3b6d425679f68ed
0506 19:56:48.435290 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.443466 (+ 8176us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.446124 (+ 2658us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.447022 (+ 898us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1204347}
I20240506 19:56:48.447458 13243 tablet_replica.cc:330] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.447639 13243 raft_consensus.cc:2229] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:48.447852 13243 pending_rounds.cc:62] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:48.447899 13243 pending_rounds.cc:68] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744071799148544 op_type: NO_OP noop_request { }
I20240506 19:56:48.448009 13243 raft_consensus.cc:2866] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:48.448083 13243 raft_consensus.cc:2258] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.448619 14051 raft_consensus.cc:482] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.448762 14051 raft_consensus.cc:504] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.448875 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 9e3d13322b4a4a2db3b6d425679f68ed (table default.Table_d9668767a7184407ac5f03e85864411e [id=e3a4e004e5c64784afdcc92d35a67b5f]) successfully deleted
I20240506 19:56:48.448997 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5a46b91cb742a994496648fbf87aa4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.449215 13120 raft_consensus.cc:2379] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:48.449391 14051 leader_election.cc:288] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.450331 12982 raft_consensus.cc:2454] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:48.450507 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3e7df09f347bfa5cad4de34dbda13" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.450738 13120 raft_consensus.cc:3037] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.450924 13244 tablet_bootstrap.cc:654] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.451184 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75467ef60504763a67fc10e8639716c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:48.451375 12982 raft_consensus.cc:3037] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.453510 13120 raft_consensus.cc:2454] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:48.453703 13242 ts_tablet_manager.cc:1888] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:48.454109 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.454281 13120 raft_consensus.cc:2454] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:48.454735 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3d13322b4a4a2db3b6d425679f68ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.454931 13124 raft_consensus.cc:2379] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:48.455209 14068 tablet.cc:1722] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:48.455583 13242 log.cc:1196] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/9c18888ae3d6420ca1de3b13fc36cf6c
I20240506 19:56:48.454690 12982 raft_consensus.cc:2454] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:48.460124 13243 ts_tablet_manager.cc:1875] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.460310 13242 ts_tablet_manager.cc:1909] T 9c18888ae3d6420ca1de3b13fc36cf6c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:48.460718 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 474) took 5683 ms. Trace:
I20240506 19:56:48.461091 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03df4632adfa4b0ab999aeaa5bb4e1f4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.461221 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378759ca310f453fba45d48879e2dd1a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.461560 13242 rpcz_store.cc:276] 0506 19:56:42.777442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:42.778066 (+ 624us) service_pool.cc:225] Handling call
0506 19:56:48.086463 (+5308397us) ts_tablet_manager.cc:1120] Deleting tablet 9c18888ae3d6420ca1de3b13fc36cf6c
0506 19:56:48.086484 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.186683 (+100199us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.425556 (+238873us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.460684 (+ 35128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5308144}
I20240506 19:56:48.461988 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 487) took 5350 ms. Trace:
W20240506 19:56:48.462967 13058 leader_election.cc:341] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.463156 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6a2c98fd374b8cbc5a65069bd4cb5b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.463874 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378759ca310f453fba45d48879e2dd1a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.464473 13242 rpcz_store.cc:276] 0506 19:56:43.111337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.111429 (+ 92us) service_pool.cc:225] Handling call
0506 19:56:48.461852 (+5350423us) ts_tablet_manager.cc:1120] Deleting tablet 378759ca310f453fba45d48879e2dd1a
0506 19:56:48.461861 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.461949 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342888}
W20240506 19:56:48.465488 12781 leader_election.cc:341] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:48.465636 12781 leader_election.cc:302] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.466081 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_73d6d75fc8f6456cba8ffeba3ecc9404"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.466367 13057 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:48.466507 13057 leader_election.cc:302] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:48.466722 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73d6d75fc8f6456cba8ffeba3ecc9404 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.466861 14069 raft_consensus.cc:2726] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.466862 13057 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.467162 13878 raft_consensus.cc:2726] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.467217 14051 raft_consensus.cc:2781] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.467272 14051 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.467335 14051 raft_consensus.cc:3037] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.467509 12922 leader_election.cc:302] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:48.467690 12922 raft_consensus.cc:2670] T 2b5a46b91cb742a994496648fbf87aa4 P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:48.468577 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b45eff5b94c4accac825233c6fa5303" } new_table_name: "default.Table_4974044201774aa6adef60247d009bd7" modify_external_catalogs: true
I20240506 19:56:48.470142 14051 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.467164 13057 leader_election.cc:302] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.472518 13057 leader_election.cc:302] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.472730 13878 raft_consensus.cc:2781] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.472848 13878 raft_consensus.cc:686] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:48.472930 14042 raft_consensus.cc:2781] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.473079 14042 raft_consensus.cc:686] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:48.473372 14042 consensus_queue.cc:237] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.473937 12922 leader_election.cc:302] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.474470 14075 raft_consensus.cc:2781] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.473076 13878 consensus_queue.cc:237] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.474602 14075 raft_consensus.cc:686] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
W20240506 19:56:48.472853 13057 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:48.474829 13057 leader_election.cc:302] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.474830 14075 consensus_queue.cc:237] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:48.475178 13057 leader_election.cc:341] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.475261 13057 leader_election.cc:302] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.475489 14076 raft_consensus.cc:2726] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240506 19:56:48.475607 13057 leader_election.cc:341] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.475690 14076 raft_consensus.cc:2726] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.475701 13057 leader_election.cc:302] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.476410 14076 raft_consensus.cc:2726] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:48.476536 13057 leader_election.cc:341] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.476621 13057 leader_election.cc:302] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.476814 14076 raft_consensus.cc:2726] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.477216 14051 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.486192 12922 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.486349 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 495) took 5329 ms. Trace:
I20240506 19:56:48.486485 14075 raft_consensus.cc:2781] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.486582 14075 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.486661 14075 raft_consensus.cc:3037] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.486550 13242 rpcz_store.cc:276] 0506 19:56:43.135576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.135641 (+ 65us) service_pool.cc:225] Handling call
0506 19:56:48.464916 (+5329275us) ts_tablet_manager.cc:1120] Deleting tablet 4b4ad903e75547fa829b82cc722fef1e
0506 19:56:48.464925 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.465009 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5325354}
I20240506 19:56:48.487402 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 494) took 5351 ms. Trace:
I20240506 19:56:48.487574 13242 rpcz_store.cc:276] 0506 19:56:43.135479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.135661 (+ 182us) service_pool.cc:225] Handling call
0506 19:56:48.487212 (+5351551us) ts_tablet_manager.cc:1120] Deleting tablet 64485c3c13e94055a019f2ef7be1f2fe
0506 19:56:48.487226 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.487360 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5343660}
I20240506 19:56:48.488101 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 497) took 5348 ms. Trace:
I20240506 19:56:48.488237 13242 rpcz_store.cc:276] 0506 19:56:43.139564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.143566 (+ 4002us) service_pool.cc:225] Handling call
0506 19:56:48.487972 (+5344406us) ts_tablet_manager.cc:1120] Deleting tablet b2a5a5e193f245de96cc82986e292f05
0506 19:56:48.487984 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.488065 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5344227}
I20240506 19:56:48.488785 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 493) took 5353 ms. Trace:
I20240506 19:56:48.488917 13242 rpcz_store.cc:276] 0506 19:56:43.135375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.135601 (+ 226us) service_pool.cc:225] Handling call
0506 19:56:48.488656 (+5353055us) ts_tablet_manager.cc:1120] Deleting tablet 116df9bd6ab04a30ba09861e765d7267
0506 19:56:48.488667 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.488749 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342992}
I20240506 19:56:48.489478 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 496) took 5350 ms. Trace:
I20240506 19:56:48.489610 13242 rpcz_store.cc:276] 0506 19:56:43.139449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.139582 (+ 133us) service_pool.cc:225] Handling call
0506 19:56:48.489257 (+5349675us) ts_tablet_manager.cc:1120] Deleting tablet ef03fcaa05d04ebd9f262f43ff7c02e4
0506 19:56:48.489268 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.489441 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5343046}
I20240506 19:56:48.489684 14075 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.490031 13243 ts_tablet_manager.cc:1888] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:48.490051 13242 tablet_replica.cc:330] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:48.490329 13243 log.cc:1196] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2417a535329341d8bb9f8218770f890d
I20240506 19:56:48.490310 13242 raft_consensus.cc:2229] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.490841 13243 ts_tablet_manager.cc:1909] T 2417a535329341d8bb9f8218770f890d P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.491101 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:48.491623 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 708) took 1261 ms. Trace:
I20240506 19:56:48.492084 13243 rpcz_store.cc:276] 0506 19:56:47.229663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.233349 (+ 3686us) service_pool.cc:225] Handling call
0506 19:56:48.447412 (+1214063us) ts_tablet_manager.cc:1120] Deleting tablet 2417a535329341d8bb9f8218770f890d
0506 19:56:48.447420 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.481100 (+ 33680us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.489539 (+ 8439us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.491607 (+ 2068us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213862}
I20240506 19:56:48.492727 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 730) took 1172 ms. Trace:
I20240506 19:56:48.493077 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2417a535329341d8bb9f8218770f890d (table default.Table_c57d5247a9fc4628ac83a4fd916f2b53 [id=0db49a8d38af446f8729cfa002006438]) successfully deleted
W20240506 19:56:48.492406 12921 leader_election.cc:341] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.493180 13243 rpcz_store.cc:276] 0506 19:56:47.320121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.579860 (+259739us) service_pool.cc:225] Handling call
0506 19:56:48.492627 (+912767us) ts_tablet_manager.cc:1120] Deleting tablet 1d53f3daaa2949a2b6883435f031dc37
0506 19:56:48.492636 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.492711 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":912579}
I20240506 19:56:48.493793 13242 raft_consensus.cc:2258] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:48.494119 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1d53f3daaa2949a2b6883435f031dc37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d53f3daaa2949a2b6883435f031dc37 already in progress: creating tablet
I20240506 19:56:48.494335 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 731) took 1163 ms. Trace:
I20240506 19:56:48.494526 13242 ts_tablet_manager.cc:1875] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.494391 14075 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.494720 12922 leader_election.cc:302] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:48.494908 12922 raft_consensus.cc:2670] T 9e3d13322b4a4a2db3b6d425679f68ed P 059a5c4a3f7e49ef95af4b8341702745: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:48.494560 13243 rpcz_store.cc:276] 0506 19:56:47.330710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580067 (+249357us) service_pool.cc:225] Handling call
0506 19:56:48.493845 (+913778us) ts_tablet_manager.cc:1120] Deleting tablet 2c96f014293242be9640515697c0733b
0506 19:56:48.493853 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.494315 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":913628}
I20240506 19:56:48.495361 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 732) took 1164 ms. Trace:
W20240506 19:56:48.495463 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2c96f014293242be9640515697c0733b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c96f014293242be9640515697c0733b already in progress: creating tablet
I20240506 19:56:48.495755 13243 rpcz_store.cc:276] 0506 19:56:47.330845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580227 (+249382us) service_pool.cc:225] Handling call
0506 19:56:48.495259 (+915032us) ts_tablet_manager.cc:1120] Deleting tablet 60d1f01c0edb4f6b8a621c1643ca1afb
0506 19:56:48.495267 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.495345 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":914903}
I20240506 19:56:48.496106 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 733) took 1165 ms. Trace:
W20240506 19:56:48.496193 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 60d1f01c0edb4f6b8a621c1643ca1afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d1f01c0edb4f6b8a621c1643ca1afb already in progress: creating tablet
I20240506 19:56:48.496183 13243 rpcz_store.cc:276] 0506 19:56:47.330948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580371 (+249423us) service_pool.cc:225] Handling call
0506 19:56:48.495996 (+915625us) ts_tablet_manager.cc:1120] Deleting tablet 1b89fb24d94042e1bfa772265a991680
0506 19:56:48.496005 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.496090 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":915502}
I20240506 19:56:48.496481 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 734) took 1031 ms. Trace:
I20240506 19:56:48.496553 13243 rpcz_store.cc:276] 0506 19:56:47.465176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580502 (+115326us) service_pool.cc:225] Handling call
0506 19:56:48.496387 (+915885us) ts_tablet_manager.cc:1120] Deleting tablet f759b560e2ab41a6a076976ccca811d1
0506 19:56:48.496394 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.496467 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":915770}
W20240506 19:56:48.497071 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f759b560e2ab41a6a076976ccca811d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f759b560e2ab41a6a076976ccca811d1 already in progress: creating tablet
I20240506 19:56:48.497983 13243 tablet_replica.cc:330] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.497874 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:48.498169 13243 raft_consensus.cc:2229] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.498311 13243 raft_consensus.cc:2258] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.498834 13243 ts_tablet_manager.cc:1875] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.499496 14080 raft_consensus.cc:482] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.499701 14080 raft_consensus.cc:504] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.500202 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.500468 14080 leader_election.cc:288] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.500562 13120 raft_consensus.cc:2379] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
I20240506 19:56:48.501365 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad527c3f2a4b53affa928e149db1df" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.501678 13058 leader_election.cc:341] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.501960 12921 leader_election.cc:341] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.502446 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:48.502699 12982 raft_consensus.cc:2379] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a9c37db9d4f41cd93820a1c81f48d4d in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:56:48.503165 12922 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.503156 13057 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.503587 14051 raft_consensus.cc:2726] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.504360 14075 raft_consensus.cc:2726] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.504582 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad527c3f2a4b53affa928e149db1df" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.504757 13120 raft_consensus.cc:2454] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:48.508598 13249 tablet_bootstrap.cc:654] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.515180 13243 ts_tablet_manager.cc:1888] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.515838 12807 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 132) took 1160 ms. Trace:
I20240506 19:56:48.516038 12807 rpcz_store.cc:276] 0506 19:56:47.355494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.355555 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:48.133335 (+777780us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x616000113e54. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4d57db00 00007f7b5128f95f 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:48.515767 (+382432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":418045824}
I20240506 19:56:48.527460 13245 tablet_bootstrap.cc:492] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.527643 13245 ts_tablet_manager.cc:1367] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.137s user 0.008s sys 0.021s
I20240506 19:56:48.528430 13245 raft_consensus.cc:348] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.528650 13245 raft_consensus.cc:374] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.528704 13245 raft_consensus.cc:729] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.528915 13245 consensus_queue.cc:260] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.529492 13245 ts_tablet_manager.cc:1398] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240506 19:56:48.529781 13245 tablet_bootstrap.cc:492] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.531817 12805 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 133) took 1005 ms. Trace:
I20240506 19:56:48.532017 12805 rpcz_store.cc:276] 0506 19:56:47.526054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.526117 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:48.132692 (+606575us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x616000113e54. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4d57db00 00007f7b5128f95f 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:48.253792 (+121100us) spinlock_profiling.cc:243] Waited 9.72 ms on lock 0x6160000f6230. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4e891628 00007f7b5128f768 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:48.531750 (+277958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":292315008}
I20240506 19:56:48.536146 13245 tablet_bootstrap.cc:654] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:48.536646 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea202c26cd154d19b6926ea28bd61844 consensus state has no leader
I20240506 19:56:48.537088 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 796628a0859c4511a8cb35e495dc8886
W20240506 19:56:48.537441 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f63e8e12fd3b453a96413f1fc769fb9f consensus state has no leader
I20240506 19:56:48.538408 14051 raft_consensus.cc:482] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.538564 14051 raft_consensus.cc:504] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.539151 14051 leader_election.cc:288] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.539588 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fb091762144f6591152680a6acd29e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.541352 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aa42760da3bc4705bb6912cc7ebbfb74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.541949 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa42760da3bc4705bb6912cc7ebbfb74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.542317 12808 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 134) took 1008 ms. Trace:
I20240506 19:56:48.542405 12808 rpcz_store.cc:276] 0506 19:56:47.534189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.534277 (+ 88us) service_pool.cc:225] Handling call
0506 19:56:48.438538 (+904261us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x616000158930. stack: 00007f7b41d3f756 00007f7b41d3f510 00007f7b40d7248d 0000000000955804 00000000009557cd 000000000095519a 00007f7b4e891628 00007f7b5128f768 00007f7b4bc1b8e6 00007f7b4bc1b615 00007f7b48a4704e 00007f7b48a45f2e 00007f7b48a4c405 00007f7b48a4cd2f 00007f7b48a4cb58 0000000000a60bcc
0506 19:56:48.542267 (+103729us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11937664}
I20240506 19:56:48.542914 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1458eb9e9352452f88190029f7dfc864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.543216 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aff3a2a3bcf74711a29fc36b829be54d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.543386 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1458eb9e9352452f88190029f7dfc864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:48.543653 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aff3a2a3bcf74711a29fc36b829be54d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.544589 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_071b75a659614286996e871c603f9dcc" } new_table_name: "default.Table_af16118ff86a4673a5e238050eb735de" modify_external_catalogs: true
I20240506 19:56:48.545500 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f8a1ac708a7d4f20bb111b87031710c9" } new_table_name: "default.Table_bf2de97d44e34eb08bd8461038259ace" modify_external_catalogs: true
I20240506 19:56:48.547279 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 796628a0859c4511a8cb35e495dc8886 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.549547 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5893f3f184b247949785e248aa0e53c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.550017 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5893f3f184b247949785e248aa0e53c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.551196 12519 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e] alter complete (version 3)
I20240506 19:56:48.551636 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_da4848522d5447739671166038af13aa" } modify_external_catalogs: true
I20240506 19:56:48.551782 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_da4848522d5447739671166038af13aa" } modify_external_catalogs: true
I20240506 19:56:48.553800 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_03fba11fcfa745868fa55d4fe134c993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.554281 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03fba11fcfa745868fa55d4fe134c993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.554873 14051 raft_consensus.cc:482] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.555073 14051 raft_consensus.cc:504] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.555877 14051 leader_election.cc:288] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.556260 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4095f9fc28ff48f68064ec8318ff7b51" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.556741 13058 leader_election.cc:341] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.565502 13058 leader_election.cc:341] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.567734 13245 tablet_bootstrap.cc:492] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.567874 13245 ts_tablet_manager.cc:1367] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.001s
I20240506 19:56:48.395105 13246 raft_consensus.cc:2229] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.568571 13245 raft_consensus.cc:348] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.568791 13245 raft_consensus.cc:374] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.568847 13245 raft_consensus.cc:729] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.569056 13245 consensus_queue.cc:260] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.569661 13245 ts_tablet_manager.cc:1398] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240506 19:56:48.570781 12507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240506 19:56:48.570833 13245 tablet_bootstrap.cc:492] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.572932 13245 tablet_bootstrap.cc:654] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.575819 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2005ef58fd4fcea16abe771ee385f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.576056 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 796628a0859c4511a8cb35e495dc8886 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.576052 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 9c18888ae3d6420ca1de3b13fc36cf6c (table default.Table_02dd5be4abb44c119d3a00fb0c4c6ea8 [id=a8abbf675b16433da6b7139648a8fb26]) successfully deleted
W20240506 19:56:48.576277 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 378759ca310f453fba45d48879e2dd1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 378759ca310f453fba45d48879e2dd1a already in progress: creating tablet
W20240506 19:56:48.577073 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4b4ad903e75547fa829b82cc722fef1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4ad903e75547fa829b82cc722fef1e already in progress: creating tablet
I20240506 19:56:48.577231 13244 tablet_bootstrap.cc:492] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:56:48.577270 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 64485c3c13e94055a019f2ef7be1f2fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64485c3c13e94055a019f2ef7be1f2fe already in progress: creating tablet
I20240506 19:56:48.577478 13244 ts_tablet_manager.cc:1367] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.191s user 0.008s sys 0.003s
W20240506 19:56:48.577481 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b2a5a5e193f245de96cc82986e292f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a5a5e193f245de96cc82986e292f05 already in progress: creating tablet
W20240506 19:56:48.577634 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 116df9bd6ab04a30ba09861e765d7267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 116df9bd6ab04a30ba09861e765d7267 already in progress: creating tablet
W20240506 19:56:48.577812 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef03fcaa05d04ebd9f262f43ff7c02e4 already in progress: creating tablet
I20240506 19:56:48.568580 13246 raft_consensus.cc:2258] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.579434 13246 ts_tablet_manager.cc:1875] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.579684 13244 raft_consensus.cc:348] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.580117 13244 raft_consensus.cc:374] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.580225 13244 raft_consensus.cc:729] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.580530 13244 consensus_queue.cc:260] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.582957 13243 log.cc:1196] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/cd57563cc6c140c184d1725a2da6a475
I20240506 19:56:48.522125 12922 leader_election.cc:302] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.538282 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 796628a0859c4511a8cb35e495dc8886 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.584468 14075 raft_consensus.cc:2781] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.584556 14075 raft_consensus.cc:482] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.584653 14075 raft_consensus.cc:3037] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.584825 12518 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:48.585227 13245 tablet_bootstrap.cc:492] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:48.585218 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b96dc180faef44d598edfa11fe65e64b consensus state has no leader
I20240506 19:56:48.585372 13245 ts_tablet_manager.cc:1367] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240506 19:56:48.586066 13245 raft_consensus.cc:348] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.586261 13245 raft_consensus.cc:374] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.586309 13245 raft_consensus.cc:729] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.580041 13249 tablet_bootstrap.cc:492] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.586521 13249 ts_tablet_manager.cc:1367] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.003s
I20240506 19:56:48.586508 13245 consensus_queue.cc:260] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.587163 13245 ts_tablet_manager.cc:1398] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:48.587145 14075 raft_consensus.cc:504] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.587414 13245 tablet_bootstrap.cc:492] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.587461 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet c5f3def738fb4cb195d08e48d7d2b293 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:48.587353 13249 raft_consensus.cc:348] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.587546 13249 raft_consensus.cc:374] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.587599 13249 raft_consensus.cc:729] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.587771 13249 consensus_queue.cc:260] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.588277 13244 ts_tablet_manager.cc:1398] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240506 19:56:48.588857 13244 tablet_bootstrap.cc:492] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.589644 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad527c3f2a4b53affa928e149db1df" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:48.588636 14075 leader_election.cc:288] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.591008 13245 tablet_bootstrap.cc:654] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.591357 13249 ts_tablet_manager.cc:1398] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:48.591729 13249 tablet_bootstrap.cc:492] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.592875 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad527c3f2a4b53affa928e149db1df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.593894 13249 tablet_bootstrap.cc:654] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.594121 13244 tablet_bootstrap.cc:654] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.598600 13246 ts_tablet_manager.cc:1888] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.598726 13246 log.cc:1196] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1372be71e1d141ed89e58cdb806431a4
I20240506 19:56:48.598994 13246 ts_tablet_manager.cc:1909] T 1372be71e1d141ed89e58cdb806431a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:48.589493 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 67399.7us
W20240506 19:56:48.590147 12921 leader_election.cc:341] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.605165 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 646) took 2533 ms. Trace:
I20240506 19:56:48.605998 13242 ts_tablet_manager.cc:1888] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.606217 13245 tablet_bootstrap.cc:492] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.606128 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad527c3f2a4b53affa928e149db1df" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.606364 13245 ts_tablet_manager.cc:1367] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240506 19:56:48.606364 13120 raft_consensus.cc:3037] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.607257 13245 raft_consensus.cc:348] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.607431 13245 raft_consensus.cc:374] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.607478 13245 raft_consensus.cc:729] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.607723 13245 consensus_queue.cc:260] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.606702 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet ca394b7f53ce445e8778dd25d0cf25c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.608229 13242 log.cc:1196] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/4a080ccac6394210b0b24b65843f9d75
I20240506 19:56:48.608851 13242 ts_tablet_manager.cc:1909] T 4a080ccac6394210b0b24b65843f9d75 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:48.609050 13245 ts_tablet_manager.cc:1398] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:48.609102 13120 raft_consensus.cc:2454] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:48.609392 13245 tablet_bootstrap.cc:492] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.611029 13246 rpcz_store.cc:276] 0506 19:56:46.072111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.072154 (+ 43us) service_pool.cc:225] Handling call
0506 19:56:48.394708 (+2322554us) ts_tablet_manager.cc:1120] Deleting tablet 1372be71e1d141ed89e58cdb806431a4
0506 19:56:48.394718 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.592159 (+197441us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.598480 (+ 6321us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.605138 (+ 6658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2322376}
I20240506 19:56:48.611456 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 647) took 2533 ms. Trace:
I20240506 19:56:48.611477 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1372be71e1d141ed89e58cdb806431a4 (table default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]) successfully deleted
I20240506 19:56:48.611555 13246 rpcz_store.cc:276] 0506 19:56:46.078452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.078515 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:48.611322 (+2532807us) ts_tablet_manager.cc:1120] Deleting tablet 7bc30f4690b24c5ea8ab2f7a26122992
0506 19:56:48.611332 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.611435 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2532609}
W20240506 19:56:48.611955 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7bc30f4690b24c5ea8ab2f7a26122992 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bc30f4690b24c5ea8ab2f7a26122992 already in progress: creating tablet
I20240506 19:56:48.612041 13245 tablet_bootstrap.cc:654] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.612289 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fb091762144f6591152680a6acd29e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.612743 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 498) took 5473 ms. Trace:
W20240506 19:56:48.612895 12507 consensus_peers.cc:485] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d -> Peer 97f390a786034cddbff757779037f861 (127.12.48.252:36867): Couldn't send request to peer 97f390a786034cddbff757779037f861. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.252:36867: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240506 19:56:48.612828 13242 rpcz_store.cc:276] 0506 19:56:43.139678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.143756 (+ 4078us) service_pool.cc:225] Handling call
0506 19:56:48.489979 (+5346223us) ts_tablet_manager.cc:1120] Deleting tablet 4a080ccac6394210b0b24b65843f9d75
0506 19:56:48.489990 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.574416 (+ 84426us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.605861 (+ 31445us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.612718 (+ 6857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5341399}
I20240506 19:56:48.613154 13249 tablet_bootstrap.cc:492] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.613402 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4095f9fc28ff48f68064ec8318ff7b51" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.613598 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 4a080ccac6394210b0b24b65843f9d75 (table default.Table_bb529dd8d37f40449f27b0e07cbc93dc [id=4b9a2f4ea4b1460f89e8981e952e0d2d]) successfully deleted
W20240506 19:56:48.613965 13057 leader_election.cc:341] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.613942 13246 tablet_replica.cc:330] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.614106 13057 leader_election.cc:302] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.614347 14072 raft_consensus.cc:2726] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.614665 13246 raft_consensus.cc:2229] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.614858 13246 raft_consensus.cc:2258] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.615067 12922 leader_election.cc:302] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.583770 13243 ts_tablet_manager.cc:1909] T cd57563cc6c140c184d1725a2da6a475 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.615535 13249 ts_tablet_manager.cc:1367] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240506 19:56:48.615712 14075 raft_consensus.cc:2781] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.615855 14075 raft_consensus.cc:686] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:48.616101 14075 consensus_queue.cc:237] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.616377 13249 raft_consensus.cc:348] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.616590 13249 raft_consensus.cc:374] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.616658 13249 raft_consensus.cc:729] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.616560 13246 ts_tablet_manager.cc:1875] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.616823 13249 consensus_queue.cc:260] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.617329 13249 ts_tablet_manager.cc:1398] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:48.619923 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 504) took 5392 ms. Trace:
I20240506 19:56:48.620014 13242 rpcz_store.cc:276] 0506 19:56:43.227153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.296423 (+ 69270us) service_pool.cc:225] Handling call
0506 19:56:48.619758 (+5323335us) ts_tablet_manager.cc:1120] Deleting tablet 31c440555cfe4195b3ae86e1f1eb6a34
0506 19:56:48.619773 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.619896 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5281993}
W20240506 19:56:48.623245 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 31c440555cfe4195b3ae86e1f1eb6a34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31c440555cfe4195b3ae86e1f1eb6a34 already in progress: creating tablet
I20240506 19:56:48.623945 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 511) took 5366 ms. Trace:
I20240506 19:56:48.624051 13242 rpcz_store.cc:276] 0506 19:56:43.257586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.407502 (+149916us) service_pool.cc:225] Handling call
0506 19:56:48.623797 (+5216295us) ts_tablet_manager.cc:1120] Deleting tablet 59b022c086d141a19a9b35c4a8c8bef9
0506 19:56:48.623809 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.623921 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5174033}
W20240506 19:56:48.624595 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 59b022c086d141a19a9b35c4a8c8bef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59b022c086d141a19a9b35c4a8c8bef9 already in progress: creating tablet
I20240506 19:56:48.624645 13249 tablet_bootstrap.cc:492] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.625157 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 522) took 4895 ms. Trace:
I20240506 19:56:48.625252 13242 rpcz_store.cc:276] 0506 19:56:43.729957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.730017 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:48.625028 (+4895011us) ts_tablet_manager.cc:1120] Deleting tablet 8a45a853e310473d844a423fbefc553a
0506 19:56:48.625042 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.625138 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4894755}
W20240506 19:56:48.625311 13057 leader_election.cc:341] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.625437 13057 leader_election.cc:302] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.625692 14042 raft_consensus.cc:2726] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.626276 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 523) took 4872 ms. Trace:
I20240506 19:56:48.626358 13242 rpcz_store.cc:276] 0506 19:56:43.753688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.755480 (+ 1792us) service_pool.cc:225] Handling call
0506 19:56:48.626164 (+4870684us) ts_tablet_manager.cc:1120] Deleting tablet 65da8b15e0c6496bb86b47c49e348af0
0506 19:56:48.626174 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.626260 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4870417}
I20240506 19:56:48.627210 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 524) took 4868 ms. Trace:
I20240506 19:56:48.627274 13242 rpcz_store.cc:276] 0506 19:56:43.758778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.761243 (+ 2465us) service_pool.cc:225] Handling call
0506 19:56:48.627114 (+4865871us) ts_tablet_manager.cc:1120] Deleting tablet 452a83470585433e8def7174300d3a7f
0506 19:56:48.627123 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.627196 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4865601}
W20240506 19:56:48.627566 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8a45a853e310473d844a423fbefc553a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a45a853e310473d844a423fbefc553a already in progress: creating tablet
W20240506 19:56:48.627830 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 65da8b15e0c6496bb86b47c49e348af0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65da8b15e0c6496bb86b47c49e348af0 already in progress: creating tablet
W20240506 19:56:48.627985 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 452a83470585433e8def7174300d3a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452a83470585433e8def7174300d3a7f already in progress: creating tablet
I20240506 19:56:48.628015 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 525) took 4865 ms. Trace:
I20240506 19:56:48.628078 13242 rpcz_store.cc:276] 0506 19:56:43.762174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.762962 (+ 788us) service_pool.cc:225] Handling call
0506 19:56:48.627916 (+4864954us) ts_tablet_manager.cc:1120] Deleting tablet 08dafb661b2443f5afadc6816868bac8
0506 19:56:48.627924 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.628002 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4864729}
I20240506 19:56:48.628536 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 735) took 1112 ms. Trace:
I20240506 19:56:48.628641 13243 rpcz_store.cc:276] 0506 19:56:47.516002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580626 (+ 64624us) service_pool.cc:225] Handling call
0506 19:56:48.496752 (+916126us) ts_tablet_manager.cc:1120] Deleting tablet cd57563cc6c140c184d1725a2da6a475
0506 19:56:48.496760 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.510004 (+ 13244us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.515049 (+ 5045us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.628516 (+113467us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10302,"tablet-delete.queue_time_us":916025}
I20240506 19:56:48.628789 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 526) took 4863 ms. Trace:
I20240506 19:56:48.628849 13242 rpcz_store.cc:276] 0506 19:56:43.765286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.766806 (+ 1520us) service_pool.cc:225] Handling call
0506 19:56:48.628699 (+4861893us) ts_tablet_manager.cc:1120] Deleting tablet 2ff3435c11854649b7daf74f3d8c33d7
0506 19:56:48.628706 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.628776 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4861659}
I20240506 19:56:48.629062 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet cd57563cc6c140c184d1725a2da6a475 (table default.Table_ff179844ac74496183bb76ddcabbcc75 [id=6a6084dba9e54d238890a1c9be17fdec]) successfully deleted
I20240506 19:56:48.629528 13242 tablet_replica.cc:330] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:48.629719 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 08dafb661b2443f5afadc6816868bac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08dafb661b2443f5afadc6816868bac8 already in progress: creating tablet
W20240506 19:56:48.629977 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2ff3435c11854649b7daf74f3d8c33d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ff3435c11854649b7daf74f3d8c33d7 already in progress: creating tablet
I20240506 19:56:48.630127 13242 raft_consensus.cc:2229] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.630270 13243 tablet_replica.cc:330] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:48.630281 13242 raft_consensus.cc:2258] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.630486 13243 raft_consensus.cc:2229] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:48.630764 13243 pending_rounds.cc:62] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:48.630816 13242 ts_tablet_manager.cc:1875] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.630821 13243 pending_rounds.cc:68] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744073115115520 op_type: NO_OP noop_request { }
I20240506 19:56:48.631002 13243 raft_consensus.cc:2866] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:48.631121 13243 raft_consensus.cc:2258] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.635125 13243 ts_tablet_manager.cc:1875] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.636868 14069 raft_consensus.cc:482] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.637022 14069 raft_consensus.cc:504] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.637879 14069 leader_election.cc:288] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.639081 13244 tablet_bootstrap.cc:492] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:56:48.639135 12783 leader_election.cc:341] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 0e2005ef58fd4fcea16abe771ee385f8
I20240506 19:56:48.639266 13244 ts_tablet_manager.cc:1367] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.003s
W20240506 19:56:48.640085 12781 leader_election.cc:341] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 0e2005ef58fd4fcea16abe771ee385f8
I20240506 19:56:48.640183 12781 leader_election.cc:302] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.640141 13244 raft_consensus.cc:348] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.640395 13244 raft_consensus.cc:374] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.640457 13244 raft_consensus.cc:729] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.640720 14069 raft_consensus.cc:2726] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.641952 13249 tablet_bootstrap.cc:654] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:48.646189 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8448ad8e9384edba7202d0bdce516a4 consensus state has no leader
I20240506 19:56:48.647655 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fac42373406c413f9eb5907032a077f9" } modify_external_catalogs: true
I20240506 19:56:48.647869 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fac42373406c413f9eb5907032a077f9" } modify_external_catalogs: true
I20240506 19:56:48.648301 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_857904c3c3954ef5a113de49478bd27f" } modify_external_catalogs: true
I20240506 19:56:48.648468 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_857904c3c3954ef5a113de49478bd27f" } modify_external_catalogs: true
I20240506 19:56:48.648521 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet a20016b56be94248905e0e8d6a35100f from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:48.649047 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_feaa9d23df4841deb38df4084445bd09" } modify_external_catalogs: true
I20240506 19:56:48.649209 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_feaa9d23df4841deb38df4084445bd09" } modify_external_catalogs: true
W20240506 19:56:48.651599 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08535fa1600b41ff919fefa93cfc4c78 consensus state has no leader
I20240506 19:56:48.651118 14042 raft_consensus.cc:482] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.652034 14042 raft_consensus.cc:504] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.652734 14080 raft_consensus.cc:482] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.652876 14080 raft_consensus.cc:504] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.653515 14080 leader_election.cc:288] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.654119 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b2a4720f6a46e4b0412a58c1e26bf7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.654328 13242 ts_tablet_manager.cc:1888] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.654433 13242 log.cc:1196] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/40ece0ad4df247eb9d380ed1c755ac99
I20240506 19:56:48.654749 13242 ts_tablet_manager.cc:1909] T 40ece0ad4df247eb9d380ed1c755ac99 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:48.654690 12921 leader_election.cc:341] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.640700 13244 consensus_queue.cc:260] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.655332 14042 leader_election.cc:288] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.655720 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b2a4720f6a46e4b0412a58c1e26bf7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.655910 13120 raft_consensus.cc:2454] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:48.655954 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9ecd2e034e49e49772db7435da3c81" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:48.656576 13057 leader_election.cc:341] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.656617 12922 leader_election.cc:302] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.657002 14075 raft_consensus.cc:2781] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.658524 14075 raft_consensus.cc:482] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.658461 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9ecd2e034e49e49772db7435da3c81" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.658900 13058 leader_election.cc:341] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.659015 13058 leader_election.cc:302] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.659287 13246 ts_tablet_manager.cc:1888] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.659332 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 527) took 4843 ms. Trace:
I20240506 19:56:48.658591 14075 raft_consensus.cc:3037] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.659482 13246 log.cc:1196] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d6e54543d1544b1c8cff3e890e787337
I20240506 19:56:48.659428 13242 rpcz_store.cc:276] 0506 19:56:43.816008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.816111 (+ 103us) service_pool.cc:225] Handling call
0506 19:56:48.629484 (+4813373us) ts_tablet_manager.cc:1120] Deleting tablet 40ece0ad4df247eb9d380ed1c755ac99
0506 19:56:48.629492 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.650350 (+ 20858us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.654224 (+ 3874us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.659305 (+ 5081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4800365}
I20240506 19:56:48.659924 13246 ts_tablet_manager.cc:1909] T d6e54543d1544b1c8cff3e890e787337 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.659936 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 528) took 4830 ms. Trace:
I20240506 19:56:48.660034 13242 rpcz_store.cc:276] 0506 19:56:43.829362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.832414 (+ 3052us) service_pool.cc:225] Handling call
0506 19:56:48.659824 (+4827410us) ts_tablet_manager.cc:1120] Deleting tablet 880ecc75bab446e39c687f3c8b740361
0506 19:56:48.659834 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.659922 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4799490}
I20240506 19:56:48.660522 13242 tablet_replica.cc:330] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:48.660686 13242 raft_consensus.cc:2229] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.661180 13242 raft_consensus.cc:2258] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.661731 13242 ts_tablet_manager.cc:1875] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.662387 14075 raft_consensus.cc:504] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.663746 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b2a4720f6a46e4b0412a58c1e26bf7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.664218 12921 leader_election.cc:341] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.664481 14075 leader_election.cc:288] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.665285 13244 ts_tablet_manager.cc:1398] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240506 19:56:48.664985 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b2a4720f6a46e4b0412a58c1e26bf7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.661154 14077 raft_consensus.cc:2726] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.665580 13120 raft_consensus.cc:3037] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.666177 13244 tablet_bootstrap.cc:492] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.668763 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 648) took 2525 ms. Trace:
I20240506 19:56:48.668980 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 00de09ddcdd948a6a3500a6c9d814c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.669339 13246 rpcz_store.cc:276] 0506 19:56:46.143007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.143060 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:48.611790 (+2468730us) ts_tablet_manager.cc:1120] Deleting tablet d6e54543d1544b1c8cff3e890e787337
0506 19:56:48.612963 (+ 1173us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.635817 (+ 22854us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.659141 (+ 23324us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.668740 (+ 9599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456239}
I20240506 19:56:48.669596 13120 raft_consensus.cc:2454] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:48.669680 13246 tablet_replica.cc:330] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.669879 13246 raft_consensus.cc:2229] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.670020 13246 raft_consensus.cc:2258] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.670024 12922 leader_election.cc:302] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.670419 14080 raft_consensus.cc:2781] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.670524 14080 raft_consensus.cc:686] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:48.670782 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 40ece0ad4df247eb9d380ed1c755ac99 (table default.Table_d91e484ef0cc4adb86af57d549a7e641 [id=9ad0be2302b94351876dba7a918a8d22]) successfully deleted
I20240506 19:56:48.670763 14080 consensus_queue.cc:237] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:48.671316 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 880ecc75bab446e39c687f3c8b740361 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 880ecc75bab446e39c687f3c8b740361 already in progress: creating tablet
I20240506 19:56:48.672395 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d6e54543d1544b1c8cff3e890e787337 (table default.Table_77ba2f5ca8614d1ebb987a154041901c [id=3045c42f00ac4ff19e0354945092c6d1]) successfully deleted
I20240506 19:56:48.675117 13249 tablet_bootstrap.cc:492] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.675630 13249 ts_tablet_manager.cc:1367] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240506 19:56:48.675734 13244 tablet_bootstrap.cc:654] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.676687 13249 raft_consensus.cc:348] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.679677 13242 ts_tablet_manager.cc:1888] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.679847 13249 raft_consensus.cc:374] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.679966 13249 raft_consensus.cc:729] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.680233 13249 consensus_queue.cc:260] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.681136 13249 ts_tablet_manager.cc:1398] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:48.681522 13249 tablet_bootstrap.cc:492] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.682987 13249 tablet_bootstrap.cc:654] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.683070 13246 ts_tablet_manager.cc:1875] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.683144 13242 log.cc:1196] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/8f66f1e46cce47639fe644fb4ed2e5c0
I20240506 19:56:48.683576 13242 ts_tablet_manager.cc:1909] T 8f66f1e46cce47639fe644fb4ed2e5c0 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:48.685123 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 538) took 4745 ms. Trace:
I20240506 19:56:48.685262 13242 rpcz_store.cc:276] 0506 19:56:43.939860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:43.939942 (+ 82us) service_pool.cc:225] Handling call
0506 19:56:48.660481 (+4720539us) ts_tablet_manager.cc:1120] Deleting tablet 8f66f1e46cce47639fe644fb4ed2e5c0
0506 19:56:48.660490 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.664911 (+ 4421us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.679553 (+ 14642us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.685099 (+ 5546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4709447}
I20240506 19:56:48.685878 13242 tablet_replica.cc:330] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:48.685940 13245 tablet_bootstrap.cc:492] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.686067 13245 ts_tablet_manager.cc:1367] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.005s
I20240506 19:56:48.686241 13242 raft_consensus.cc:2229] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.686440 13242 raft_consensus.cc:2258] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.686993 13242 ts_tablet_manager.cc:1875] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.691606 13885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 19:56:48.687234 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 8f66f1e46cce47639fe644fb4ed2e5c0 (table default.Table_ca61f3a9aefa4fba8fea32d717de3fb1 [id=bf75c039ad324e50a2ccb889439e2b4c]) successfully deleted
I20240506 19:56:48.691965 14082 raft_consensus.cc:482] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.692121 14082 raft_consensus.cc:504] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.692940 14082 leader_election.cc:288] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.693696 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9ecd2e034e49e49772db7435da3c81" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.686851 13245 raft_consensus.cc:348] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.694098 13245 raft_consensus.cc:374] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:48.694293 12921 leader_election.cc:341] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.694406 13245 raft_consensus.cc:729] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.694563 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:48.694689 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9ecd2e034e49e49772db7435da3c81" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.694881 13120 raft_consensus.cc:2454] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:48.694859 13245 consensus_queue.cc:260] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.695394 13245 ts_tablet_manager.cc:1398] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:48.695498 12922 leader_election.cc:302] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.695755 14082 raft_consensus.cc:2781] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.695820 14082 raft_consensus.cc:482] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.695869 14082 raft_consensus.cc:3037] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.698141 13244 tablet_bootstrap.cc:492] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:48.698333 13244 ts_tablet_manager.cc:1367] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20240506 19:56:48.698700 14082 raft_consensus.cc:504] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.699609 13244 raft_consensus.cc:348] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.699874 13249 tablet_bootstrap.cc:492] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.699913 13244 raft_consensus.cc:374] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.699734 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9ecd2e034e49e49772db7435da3c81" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:48.699981 13244 raft_consensus.cc:729] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.700094 13249 ts_tablet_manager.cc:1367] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240506 19:56:48.700359 13244 consensus_queue.cc:260] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:48.700510 12921 leader_election.cc:341] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.700870 14082 leader_election.cc:288] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.700990 13244 ts_tablet_manager.cc:1398] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:48.701268 13244 tablet_bootstrap.cc:492] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.702396 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 19:56:48.703372 13244 tablet_bootstrap.cc:654] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.704528 13243 ts_tablet_manager.cc:1888] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:48.704747 13245 tablet_bootstrap.cc:492] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.705749 13243 log.cc:1196] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d8f3e7df09f347bfa5cad4de34dbda13
I20240506 19:56:48.705837 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9ecd2e034e49e49772db7435da3c81" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:48.706040 13120 raft_consensus.cc:3037] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.706163 13243 ts_tablet_manager.cc:1909] T d8f3e7df09f347bfa5cad4de34dbda13 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.706589 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 19:56:48.708377 13245 tablet_bootstrap.cc:654] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.711601 13249 raft_consensus.cc:348] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.711902 13249 raft_consensus.cc:374] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.712018 13249 raft_consensus.cc:729] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.712296 13249 consensus_queue.cc:260] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.713034 13249 ts_tablet_manager.cc:1398] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240506 19:56:48.716514 13242 ts_tablet_manager.cc:1888] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.716694 13242 log.cc:1196] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b38453bf678748399339806d12993c88
I20240506 19:56:48.717056 13242 ts_tablet_manager.cc:1909] T b38453bf678748399339806d12993c88 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:48.717895 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 551) took 4580 ms. Trace:
I20240506 19:56:48.718036 13242 rpcz_store.cc:276] 0506 19:56:44.137313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.137393 (+ 80us) service_pool.cc:225] Handling call
0506 19:56:48.685829 (+4548436us) ts_tablet_manager.cc:1120] Deleting tablet b38453bf678748399339806d12993c88
0506 19:56:48.685838 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.694965 (+ 9127us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.716390 (+ 21425us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.717872 (+ 1482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4536370}
I20240506 19:56:48.718680 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b38453bf678748399339806d12993c88 (table default.Table_62612e9478664e9d86a80129badd6c12 [id=8f671c29345e43b6ba8379f8182cba65]) successfully deleted
I20240506 19:56:48.718858 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 554) took 4567 ms. Trace:
I20240506 19:56:48.718972 13242 rpcz_store.cc:276] 0506 19:56:44.151154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.153673 (+ 2519us) service_pool.cc:225] Handling call
0506 19:56:48.718673 (+4565000us) ts_tablet_manager.cc:1120] Deleting tablet d19db9cff74843e087fd82db2475c29c
0506 19:56:48.718683 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.718780 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4563830}
W20240506 19:56:48.719447 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d19db9cff74843e087fd82db2475c29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d19db9cff74843e087fd82db2475c29c already in progress: creating tablet
I20240506 19:56:48.719590 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 550) took 4583 ms. Trace:
I20240506 19:56:48.719698 13242 rpcz_store.cc:276] 0506 19:56:44.136351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.139870 (+ 3519us) service_pool.cc:225] Handling call
0506 19:56:48.719418 (+4579548us) ts_tablet_manager.cc:1120] Deleting tablet 986f9c83b04f41b49e9b78fce731a90f
0506 19:56:48.719427 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.719513 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4562587}
W20240506 19:56:48.720083 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 986f9c83b04f41b49e9b78fce731a90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 986f9c83b04f41b49e9b78fce731a90f already in progress: creating tablet
I20240506 19:56:48.720391 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 552) took 4572 ms. Trace:
I20240506 19:56:48.720518 13242 rpcz_store.cc:276] 0506 19:56:44.148417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.149033 (+ 616us) service_pool.cc:225] Handling call
0506 19:56:48.720078 (+4571045us) ts_tablet_manager.cc:1120] Deleting tablet 40b9731a473449b19a9e95653a9d0830
0506 19:56:48.720086 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.720375 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4546816}
I20240506 19:56:48.720988 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 553) took 4570 ms. Trace:
I20240506 19:56:48.721110 13242 rpcz_store.cc:276] 0506 19:56:44.151020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.151258 (+ 238us) service_pool.cc:225] Handling call
0506 19:56:48.720897 (+4569639us) ts_tablet_manager.cc:1120] Deleting tablet c652feb405ef4f1cab8ee602e9d3eb18
0506 19:56:48.720905 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.720974 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4543784}
I20240506 19:56:48.721621 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 555) took 4563 ms. Trace:
I20240506 19:56:48.721761 13242 rpcz_store.cc:276] 0506 19:56:44.157861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.161275 (+ 3414us) service_pool.cc:225] Handling call
0506 19:56:48.721512 (+4560237us) ts_tablet_manager.cc:1120] Deleting tablet ee62b918bba543a6b3e467fd8dc59878
0506 19:56:48.721523 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.721603 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355467}
I20240506 19:56:48.722317 13242 tablet_replica.cc:330] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:48.722319 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 40b9731a473449b19a9e95653a9d0830 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40b9731a473449b19a9e95653a9d0830 already in progress: creating tablet
W20240506 19:56:48.722560 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c652feb405ef4f1cab8ee602e9d3eb18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c652feb405ef4f1cab8ee602e9d3eb18 already in progress: creating tablet
I20240506 19:56:48.722606 13242 raft_consensus.cc:2229] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:48.722774 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ee62b918bba543a6b3e467fd8dc59878 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee62b918bba543a6b3e467fd8dc59878 already in progress: creating tablet
I20240506 19:56:48.722920 13242 raft_consensus.cc:2258] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.723534 13242 ts_tablet_manager.cc:1875] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.708621 13120 raft_consensus.cc:2454] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:48.724664 12922 leader_election.cc:302] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.725087 14080 raft_consensus.cc:2781] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.725214 14080 raft_consensus.cc:686] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:48.725508 14080 consensus_queue.cc:237] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.727195 13249 tablet_bootstrap.cc:492] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.727216 13246 ts_tablet_manager.cc:1888] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.727317 13246 log.cc:1196] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:48.727602 13246 ts_tablet_manager.cc:1909] T ef03fcaa05d04ebd9f262f43ff7c02e4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.727982 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8689ec0a8e134bceb010053b1f5a343d
I20240506 19:56:48.729033 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 8689ec0a8e134bceb010053b1f5a343d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.729425 13249 tablet_bootstrap.cc:654] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.730969 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 8689ec0a8e134bceb010053b1f5a343d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.731235 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 649) took 2467 ms. Trace:
I20240506 19:56:48.731832 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 8689ec0a8e134bceb010053b1f5a343d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.732121 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet a8448ad8e9384edba7202d0bdce516a4 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:48.731326 13246 rpcz_store.cc:276] 0506 19:56:46.263495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.263551 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:48.669633 (+2406082us) ts_tablet_manager.cc:1120] Deleting tablet ef03fcaa05d04ebd9f262f43ff7c02e4
0506 19:56:48.669644 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.704071 (+ 34427us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.727110 (+ 23039us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.731209 (+ 4099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2405896}
I20240506 19:56:48.733889 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 651) took 2368 ms. Trace:
I20240506 19:56:48.733942 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ef03fcaa05d04ebd9f262f43ff7c02e4 (table default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]) successfully deleted
I20240506 19:56:48.733978 13246 rpcz_store.cc:276] 0506 19:56:46.365258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.365343 (+ 85us) service_pool.cc:225] Handling call
0506 19:56:48.733749 (+2368406us) ts_tablet_manager.cc:1120] Deleting tablet 1750c4dd8cdc40509c553b63d166c466
0506 19:56:48.733762 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.733870 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368190}
I20240506 19:56:48.734308 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 650) took 2369 ms. Trace:
I20240506 19:56:48.734390 13246 rpcz_store.cc:276] 0506 19:56:46.365150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.365589 (+ 439us) service_pool.cc:225] Handling call
0506 19:56:48.734215 (+2368626us) ts_tablet_manager.cc:1120] Deleting tablet 42151ea2c1214c149d7dcac9a2c447aa
0506 19:56:48.734223 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.734294 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368518}
I20240506 19:56:48.734681 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 652) took 2367 ms. Trace:
I20240506 19:56:48.734755 13246 rpcz_store.cc:276] 0506 19:56:46.367575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.367756 (+ 181us) service_pool.cc:225] Handling call
0506 19:56:48.734592 (+2366836us) ts_tablet_manager.cc:1120] Deleting tablet a20016b56be94248905e0e8d6a35100f
0506 19:56:48.734600 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.734668 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2366674}
I20240506 19:56:48.729449 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 736) took 1200 ms. Trace:
I20240506 19:56:48.735044 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 653) took 2367 ms. Trace:
I20240506 19:56:48.735105 13246 rpcz_store.cc:276] 0506 19:56:46.367708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.367957 (+ 249us) service_pool.cc:225] Handling call
0506 19:56:48.734952 (+2366995us) ts_tablet_manager.cc:1120] Deleting tablet 028055b9df1f4667bd2e54909690e327
0506 19:56:48.734959 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.735030 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2366876}
I20240506 19:56:48.735105 13243 rpcz_store.cc:276] 0506 19:56:47.529077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580726 (+ 51649us) service_pool.cc:225] Handling call
0506 19:56:48.630221 (+1049495us) ts_tablet_manager.cc:1120] Deleting tablet d8f3e7df09f347bfa5cad4de34dbda13
0506 19:56:48.630231 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.679924 (+ 49693us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.704440 (+ 24516us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.729428 (+ 24988us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049385}
I20240506 19:56:48.735580 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d8f3e7df09f347bfa5cad4de34dbda13 (table default.Table_d5bb6a8c548e46ecb59035f49816ad52 [id=70785b5175b34eb68dc3fcba2255645b]) successfully deleted
I20240506 19:56:48.736219 13243 tablet_replica.cc:330] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:48.736284 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1750c4dd8cdc40509c553b63d166c466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1750c4dd8cdc40509c553b63d166c466 already in progress: creating tablet
I20240506 19:56:48.736542 13243 raft_consensus.cc:2229] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:48.736634 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 42151ea2c1214c149d7dcac9a2c447aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42151ea2c1214c149d7dcac9a2c447aa already in progress: creating tablet
I20240506 19:56:48.736774 13243 raft_consensus.cc:2258] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:48.736883 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a20016b56be94248905e0e8d6a35100f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20016b56be94248905e0e8d6a35100f already in progress: creating tablet
W20240506 19:56:48.737100 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 028055b9df1f4667bd2e54909690e327 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 028055b9df1f4667bd2e54909690e327 already in progress: creating tablet
I20240506 19:56:48.737519 13243 ts_tablet_manager.cc:1875] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.737754 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_68a353433c044a28b362eb7d3606bfb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.738503 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68a353433c044a28b362eb7d3606bfb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.741854 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 654) took 2357 ms. Trace:
I20240506 19:56:48.741973 13246 rpcz_store.cc:276] 0506 19:56:46.377599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.377658 (+ 59us) service_pool.cc:225] Handling call
0506 19:56:48.735306 (+2357648us) ts_tablet_manager.cc:1120] Deleting tablet a2e061a5a97646cc9b14ddf741c5b88e
0506 19:56:48.735314 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.735387 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2357457}
I20240506 19:56:48.742376 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 655) took 2348 ms. Trace:
I20240506 19:56:48.742463 13246 rpcz_store.cc:276] 0506 19:56:46.393853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.394062 (+ 209us) service_pool.cc:225] Handling call
0506 19:56:48.742246 (+2348184us) ts_tablet_manager.cc:1120] Deleting tablet bb04d06a2b084079ac6357415b124752
0506 19:56:48.742258 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.742358 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347999}
I20240506 19:56:48.742789 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 656) took 2348 ms. Trace:
I20240506 19:56:48.742651 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f76cd2f478bd458697b5873a16fbb55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.742857 13246 rpcz_store.cc:276] 0506 19:56:46.393975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.394275 (+ 300us) service_pool.cc:225] Handling call
0506 19:56:48.742692 (+2348417us) ts_tablet_manager.cc:1120] Deleting tablet cd6245e4adce4038ad6f988877d75045
0506 19:56:48.742700 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.742772 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2348294}
I20240506 19:56:48.743114 13246 tablet_replica.cc:330] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:48.743230 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f76cd2f478bd458697b5873a16fbb55b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.743253 13249 tablet_bootstrap.cc:492] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.743295 13246 raft_consensus.cc:2229] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.743448 13246 raft_consensus.cc:2258] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.743474 13249 ts_tablet_manager.cc:1367] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20240506 19:56:48.743981 13246 ts_tablet_manager.cc:1875] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.744550 13249 raft_consensus.cc:348] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.744787 13249 raft_consensus.cc:374] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.744874 13249 raft_consensus.cc:729] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:48.744994 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a2e061a5a97646cc9b14ddf741c5b88e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e061a5a97646cc9b14ddf741c5b88e already in progress: creating tablet
I20240506 19:56:48.745198 13249 consensus_queue.cc:260] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.745867 13249 ts_tablet_manager.cc:1398] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:48.746378 13249 tablet_bootstrap.cc:492] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.749050 13242 ts_tablet_manager.cc:1888] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:48.749178 13242 log.cc:1196] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/acd3fc62e9614cf3bde0202ee6832bb3
I20240506 19:56:48.749567 13242 ts_tablet_manager.cc:1909] T acd3fc62e9614cf3bde0202ee6832bb3 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:48.750381 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 559) took 4362 ms. Trace:
I20240506 19:56:48.750486 13242 rpcz_store.cc:276] 0506 19:56:44.388383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.388441 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:48.722274 (+4333833us) ts_tablet_manager.cc:1120] Deleting tablet acd3fc62e9614cf3bde0202ee6832bb3
0506 19:56:48.722282 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.746076 (+ 23794us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.748938 (+ 2862us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.750357 (+ 1419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4333370}
W20240506 19:56:48.745225 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bb04d06a2b084079ac6357415b124752 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb04d06a2b084079ac6357415b124752 already in progress: creating tablet
W20240506 19:56:48.751505 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet cd6245e4adce4038ad6f988877d75045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6245e4adce4038ad6f988877d75045 already in progress: creating tablet
I20240506 19:56:48.750998 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet acd3fc62e9614cf3bde0202ee6832bb3 (table default.Table_291e179ddb654a8e90d5e5ea4db36e60 [id=c6e68683ddb542be99a54eea5e7a6ba4]) successfully deleted
I20240506 19:56:48.752246 13244 tablet_bootstrap.cc:492] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:48.752411 13244 ts_tablet_manager.cc:1367] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240506 19:56:48.752702 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c5a00a4676e54afab4a6a801cb898192" } new_table_name: "default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5" modify_external_catalogs: true
I20240506 19:56:48.753100 13244 raft_consensus.cc:348] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.753276 13244 raft_consensus.cc:374] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.753346 13244 raft_consensus.cc:729] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.753559 13244 consensus_queue.cc:260] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.754209 13245 tablet_bootstrap.cc:492] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.754120 13244 ts_tablet_manager.cc:1398] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:48.754359 13245 ts_tablet_manager.cc:1367] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.004s
I20240506 19:56:48.755117 13245 raft_consensus.cc:348] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.755246 13244 tablet_bootstrap.cc:492] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.755312 13245 raft_consensus.cc:374] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.755374 13245 raft_consensus.cc:729] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.751132 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:48.755612 13245 consensus_queue.cc:260] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.756124 13245 ts_tablet_manager.cc:1398] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:48.756397 13245 tablet_bootstrap.cc:492] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.760804 13244 tablet_bootstrap.cc:654] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.761142 13249 tablet_bootstrap.cc:654] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.765090 13249 tablet_bootstrap.cc:492] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.765358 13249 ts_tablet_manager.cc:1367] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240506 19:56:48.765971 12518 catalog_manager.cc:5484] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:48.766422 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:48.766224 13249 raft_consensus.cc:348] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.766675 13249 raft_consensus.cc:374] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.766695 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]). Expected version 3 got 0
I20240506 19:56:48.766822 13249 raft_consensus.cc:729] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.767148 13249 consensus_queue.cc:260] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.767820 13249 ts_tablet_manager.cc:1398] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:48.768082 12733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8689ec0a8e134bceb010053b1f5a343d
I20240506 19:56:48.768221 13249 tablet_bootstrap.cc:492] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.769765 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9e17d0a261734d45866525b35de7cba3
I20240506 19:56:48.770135 13249 tablet_bootstrap.cc:654] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.770754 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e17d0a261734d45866525b35de7cba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.771304 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 562) took 4320 ms. Trace:
I20240506 19:56:48.771396 13242 rpcz_store.cc:276] 0506 19:56:44.451326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.451513 (+ 187us) service_pool.cc:225] Handling call
0506 19:56:48.771142 (+4319629us) ts_tablet_manager.cc:1120] Deleting tablet 1e22e5a015f74526b49363bc789cb4ed
0506 19:56:48.771155 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.771279 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4319404}
I20240506 19:56:48.771793 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e17d0a261734d45866525b35de7cba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.771896 13242 tablet_replica.cc:330] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:48.772063 13242 raft_consensus.cc:2229] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.772361 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_001c9250e65b432f8e9d4c1a38bda161" } new_table_name: "default.Table_7fa5f678736244999b82beddbbb9ddb5" modify_external_catalogs: true
I20240506 19:56:48.773222 13246 ts_tablet_manager.cc:1888] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.774201 13242 raft_consensus.cc:2258] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.774935 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c5f3def738fb4cb195d08e48d7d2b293
I20240506 19:56:48.774981 13242 ts_tablet_manager.cc:1875] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.775449 12522 catalog_manager.cc:6554] ReplaceTablet: tablet c5f3def738fb4cb195d08e48d7d2b293 deleted and replaced by tablet 98571fa2168644969c851a8edd343a7a
I20240506 19:56:48.775720 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet c5f3def738fb4cb195d08e48d7d2b293 with delete_type TABLET_DATA_DELETED (replaced by tablet 98571fa2168644969c851a8edd343a7a) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:48.777782 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1e22e5a015f74526b49363bc789cb4ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e22e5a015f74526b49363bc789cb4ed already in progress: creating tablet
I20240506 19:56:48.778185 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e17d0a261734d45866525b35de7cba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.778409 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c1590e36b4471d8e72f4b07410ee0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.778633 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet c5f3def738fb4cb195d08e48d7d2b293 with delete_type TABLET_DATA_DELETED (replaced by tablet 98571fa2168644969c851a8edd343a7a) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.777236 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b33a15919aac48e699db676197ad1e75" } modify_external_catalogs: true
I20240506 19:56:48.780323 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b33a15919aac48e699db676197ad1e75" } modify_external_catalogs: true
I20240506 19:56:48.781759 13244 tablet_bootstrap.cc:492] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:48.781891 13244 ts_tablet_manager.cc:1367] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20240506 19:56:48.782724 13243 ts_tablet_manager.cc:1888] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.782702 13244 raft_consensus.cc:348] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.782891 13244 raft_consensus.cc:374] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.782935 13244 raft_consensus.cc:729] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.783138 13244 consensus_queue.cc:260] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.783413 13243 log.cc:1196] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b2a5a5e193f245de96cc82986e292f05
I20240506 19:56:48.783694 13244 ts_tablet_manager.cc:1398] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:48.783989 13244 tablet_bootstrap.cc:492] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.756099 12520 catalog_manager.cc:5484] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:48.784816 13243 ts_tablet_manager.cc:1909] T b2a5a5e193f245de96cc82986e292f05 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:48.784785 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 3 got 0
I20240506 19:56:48.785447 13244 tablet_bootstrap.cc:654] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.788389 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 737) took 1259 ms. Trace:
I20240506 19:56:48.789075 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 8689ec0a8e134bceb010053b1f5a343d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.790959 12966 tablet_service.cc:1467] Processing CreateTablet for tablet e26d7da25ce1408a8e2cd46ba6082483 (DEFAULT_TABLE table=default.Table_a1625cd2cd914b31b16e4b86b2aa545d [id=405ec704b2af43628e19824582b94eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.792115 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26d7da25ce1408a8e2cd46ba6082483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.793010 12961 tablet_service.cc:1467] Processing CreateTablet for tablet b3579b5aee2340c7a41b709f659abbfa (DEFAULT_TABLE table=default.Table_5638dbb8bb784639b338a3f8d5a1b492 [id=b5cf40b4c8a948fa94b55af76ae85e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.793380 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3579b5aee2340c7a41b709f659abbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.793882 12811 tablet_service.cc:1467] Processing CreateTablet for tablet e26d7da25ce1408a8e2cd46ba6082483 (DEFAULT_TABLE table=default.Table_a1625cd2cd914b31b16e4b86b2aa545d [id=405ec704b2af43628e19824582b94eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.794235 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26d7da25ce1408a8e2cd46ba6082483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.797446 12823 tablet_service.cc:1467] Processing CreateTablet for tablet b3579b5aee2340c7a41b709f659abbfa (DEFAULT_TABLE table=default.Table_5638dbb8bb784639b338a3f8d5a1b492 [id=b5cf40b4c8a948fa94b55af76ae85e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.797811 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3579b5aee2340c7a41b709f659abbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.793526 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 8689ec0a8e134bceb010053b1f5a343d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.801858 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 4d12edf17784486b9c598c2f29675060 (DEFAULT_TABLE table=default.Table_f9f5211355fa4a10b23a184e7c55ae7c [id=e311f1c887124c15a887b9def426858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.802235 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d12edf17784486b9c598c2f29675060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.803339 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ce6483470c864697be6d9590ec95edfc
I20240506 19:56:48.805455 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e1d7a850a3e644c3a38ab224efcc8af5
I20240506 19:56:48.806706 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6483470c864697be6d9590ec95edfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.806780 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d7a850a3e644c3a38ab224efcc8af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.809475 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a8448ad8e9384edba7202d0bdce516a4
I20240506 19:56:48.810312 12519 catalog_manager.cc:6554] ReplaceTablet: tablet a8448ad8e9384edba7202d0bdce516a4 deleted and replaced by tablet 0a6db7d0ad024f53ad76920bcbc0248e
I20240506 19:56:48.810384 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d7a850a3e644c3a38ab224efcc8af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.810425 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet a8448ad8e9384edba7202d0bdce516a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0a6db7d0ad024f53ad76920bcbc0248e) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.810603 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6483470c864697be6d9590ec95edfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.811465 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet a8448ad8e9384edba7202d0bdce516a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0a6db7d0ad024f53ad76920bcbc0248e) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.813872 13243 rpcz_store.cc:276] 0506 19:56:47.529216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.580839 (+ 51623us) service_pool.cc:225] Handling call
0506 19:56:48.735923 (+1155084us) ts_tablet_manager.cc:1120] Deleting tablet b2a5a5e193f245de96cc82986e292f05
0506 19:56:48.735934 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.766494 (+ 30560us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.782609 (+ 16115us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.788365 (+ 5756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154975}
I20240506 19:56:48.814311 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b2a5a5e193f245de96cc82986e292f05 (table default.Table_384bed2d6d984f9d88c62381e38b6321 [id=f83571d593d64e58876888bfb31ccb28]) successfully deleted
W20240506 19:56:48.815011 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a8e331dc24e4d06bf81c8a4b7d70156 already in progress: creating tablet
I20240506 19:56:48.816071 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a6709e9c0a394e43a00065cca48b8cdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.816720 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6709e9c0a394e43a00065cca48b8cdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.774266 13245 tablet_bootstrap.cc:654] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.776227 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet c5f3def738fb4cb195d08e48d7d2b293 with delete_type TABLET_DATA_DELETED (replaced by tablet 98571fa2168644969c851a8edd343a7a) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.790001 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 8689ec0a8e134bceb010053b1f5a343d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.792558 13098 tablet_service.cc:1467] Processing CreateTablet for tablet e26d7da25ce1408a8e2cd46ba6082483 (DEFAULT_TABLE table=default.Table_a1625cd2cd914b31b16e4b86b2aa545d [id=405ec704b2af43628e19824582b94eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.801136 13096 tablet_service.cc:1467] Processing CreateTablet for tablet b3579b5aee2340c7a41b709f659abbfa (DEFAULT_TABLE table=default.Table_5638dbb8bb784639b338a3f8d5a1b492 [id=b5cf40b4c8a948fa94b55af76ae85e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.812592 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d7a850a3e644c3a38ab224efcc8af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.818286 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26d7da25ce1408a8e2cd46ba6082483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.812660 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6483470c864697be6d9590ec95edfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.812798 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet a8448ad8e9384edba7202d0bdce516a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0a6db7d0ad024f53ad76920bcbc0248e) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.818599 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3579b5aee2340c7a41b709f659abbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.819108 13244 tablet_bootstrap.cc:492] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:48.819231 13244 ts_tablet_manager.cc:1367] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.006s
I20240506 19:56:48.819525 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet 4be929f0e72e4d22a9af2fe6b3072bbe from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:48.819885 13244 raft_consensus.cc:348] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.820101 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_86cd3e5b876348eca4cd2b79f045de74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.820633 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86cd3e5b876348eca4cd2b79f045de74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.820067 13244 raft_consensus.cc:374] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.821198 13244 raft_consensus.cc:729] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.821692 13244 consensus_queue.cc:260] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.822706 13244 ts_tablet_manager.cc:1398] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:48.823002 13244 tablet_bootstrap.cc:492] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:48.826535 13246 log.cc:1196] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/88dba275893f4696a5318ec21f0b2477
I20240506 19:56:48.826941 13246 ts_tablet_manager.cc:1909] T 88dba275893f4696a5318ec21f0b2477 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.827036 13244 tablet_bootstrap.cc:654] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.828023 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 658) took 2428 ms. Trace:
I20240506 19:56:48.828176 13246 rpcz_store.cc:276] 0506 19:56:46.399596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.399810 (+ 214us) service_pool.cc:225] Handling call
0506 19:56:48.743073 (+2343263us) ts_tablet_manager.cc:1120] Deleting tablet 88dba275893f4696a5318ec21f0b2477
0506 19:56:48.743081 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.770212 (+ 27131us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.773129 (+ 2917us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.828005 (+ 54876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343029}
I20240506 19:56:48.828579 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:48.828886 13246 tablet_replica.cc:330] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.829141 13246 raft_consensus.cc:2229] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.829263 13246 raft_consensus.cc:2258] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.829877 14077 raft_consensus.cc:482] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.830024 14077 raft_consensus.cc:504] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.830651 14077 leader_election.cc:288] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.828685 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 88dba275893f4696a5318ec21f0b2477 (table default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 [id=194197559e4e4469b5a3622465e90f2e]) successfully deleted
I20240506 19:56:48.830976 13245 tablet_bootstrap.cc:492] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.829005 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6c86bd8e4c954a06a24af7379533208a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.831236 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb90bf767c94d73ace6411cfd31d65c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.831624 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb90bf767c94d73ace6411cfd31d65c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:48.831836 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c86bd8e4c954a06a24af7379533208a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.831104 13245 ts_tablet_manager.cc:1367] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.005s
W20240506 19:56:48.832489 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet df04fb90230d4bf4a060db993adafd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df04fb90230d4bf4a060db993adafd14 already in progress: creating tablet
I20240506 19:56:48.829593 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7ec0af39e86c4ffeb0622785e6a817be" } new_table_name: "default.Table_c64bdf2c9da94cd3ace74befeae06a29" modify_external_catalogs: true
I20240506 19:56:48.833027 13245 raft_consensus.cc:348] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.833225 13245 raft_consensus.cc:374] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:48.833250 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8eb9296bb1104b4cbe279eecd3683796 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb9296bb1104b4cbe279eecd3683796 already in progress: creating tablet
I20240506 19:56:48.833320 13245 raft_consensus.cc:729] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.833508 13245 consensus_queue.cc:260] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.834038 13245 ts_tablet_manager.cc:1398] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240506 19:56:48.834128 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet cc22261ab2034e818834ba97c2760dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc22261ab2034e818834ba97c2760dad already in progress: creating tablet
I20240506 19:56:48.834301 13245 tablet_bootstrap.cc:492] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.834309 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8335f9308046279a25a4a95d8e0f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.835793 13245 tablet_bootstrap.cc:654] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:48.838837 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bd3130bc41084bfaabaaecc0985e378d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd3130bc41084bfaabaaecc0985e378d already in progress: creating tablet
W20240506 19:56:48.831743 13058 leader_election.cc:341] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.839730 13057 leader_election.cc:341] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.839951 13057 leader_election.cc:302] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.840370 14077 raft_consensus.cc:2726] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.842711 13245 tablet_bootstrap.cc:492] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.842875 13245 ts_tablet_manager.cc:1367] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:56:48.843355 14080 raft_consensus.cc:482] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.843523 14080 raft_consensus.cc:504] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.843636 13245 raft_consensus.cc:348] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.843847 13245 raft_consensus.cc:374] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.843920 13245 raft_consensus.cc:729] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.844192 13245 consensus_queue.cc:260] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.844513 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03df4632adfa4b0ab999aeaa5bb4e1f4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.844743 13245 ts_tablet_manager.cc:1398] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:48.845052 13245 tablet_bootstrap.cc:492] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:48.845214 12921 leader_election.cc:341] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.845714 14080 leader_election.cc:288] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.846407 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03df4632adfa4b0ab999aeaa5bb4e1f4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.846637 13120 raft_consensus.cc:2454] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:48.846746 13249 tablet_bootstrap.cc:492] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:48.846896 13249 ts_tablet_manager.cc:1367] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20240506 19:56:48.846925 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d (DEFAULT_TABLE table=default.Table_28fc76ba360b4dd4b805761b13f00f04 [id=f43d09fdf4ff4890a6fed18acf484153]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.847354 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.847633 12922 leader_election.cc:302] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.847963 13249 raft_consensus.cc:348] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.848174 13249 raft_consensus.cc:374] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.848246 13249 raft_consensus.cc:729] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.848524 13249 consensus_queue.cc:260] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.848695 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 5de5e3db2bf34d6ab23b58d552d2f800 (DEFAULT_TABLE table=default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.848829 14082 raft_consensus.cc:2781] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.848912 14082 raft_consensus.cc:482] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.849007 14082 raft_consensus.cc:3037] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.849270 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d (DEFAULT_TABLE table=default.Table_28fc76ba360b4dd4b805761b13f00f04 [id=f43d09fdf4ff4890a6fed18acf484153]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.851384 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 5de5e3db2bf34d6ab23b58d552d2f800 (DEFAULT_TABLE table=default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.851771 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de5e3db2bf34d6ab23b58d552d2f800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.852190 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de5e3db2bf34d6ab23b58d552d2f800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.852746 14082 raft_consensus.cc:504] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.831638 13246 ts_tablet_manager.cc:1875] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.855937 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03df4632adfa4b0ab999aeaa5bb4e1f4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:48.857615 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd26e6b8115644cabe6532c7be1bbe95 consensus state has no leader
I20240506 19:56:48.859438 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4f382a89f94e4fdf9b8266d049635329
I20240506 19:56:48.860050 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03df4632adfa4b0ab999aeaa5bb4e1f4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:48.860167 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e02577dc6c2049a1abb47ed0fe729e18 consensus state has no leader
I20240506 19:56:48.860262 13120 raft_consensus.cc:3037] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.845996 14042 raft_consensus.cc:482] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.861827 14042 raft_consensus.cc:504] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.862437 14042 leader_election.cc:288] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:48.862756 13120 raft_consensus.cc:2454] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:48.862862 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_eb33ab4810b94486b70e278e548b02ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.863348 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 034d6309de0740d0a5806b3847e334cb with delete_type TABLET_DATA_DELETED (replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:48.863449 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb33ab4810b94486b70e278e548b02ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.863555 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f382a89f94e4fdf9b8266d049635329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:48.863624 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet bafd3c4706f74e7da36a8f3538760f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.863833 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f382a89f94e4fdf9b8266d049635329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.864006 13245 tablet_bootstrap.cc:654] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.864240 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:48.864780 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1104e7d94ecc44c2917aced82c7f7baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.864926 13058 leader_election.cc:341] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.865270 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1104e7d94ecc44c2917aced82c7f7baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.866874 13246 ts_tablet_manager.cc:1888] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.867002 13246 log.cc:1196] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/104337875517445f862d82ece3a84889
I20240506 19:56:48.867210 14082 leader_election.cc:288] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.867432 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:48.868096 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f382a89f94e4fdf9b8266d049635329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.867985 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1799610d365d46859579977132bedf50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:48.868157 12963 tablet_service.cc:1467] Processing CreateTablet for tablet fbcbeb22bfce436d931ff274248500fd (DEFAULT_TABLE table=default.Table_ecc91044eebb4ad6b2799b1569836c8c [id=5f96354a891c42e6aed78d7cbb5d6f00]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.868489 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1799610d365d46859579977132bedf50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.868736 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 5de5e3db2bf34d6ab23b58d552d2f800 (DEFAULT_TABLE table=default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.872263 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5b7eae44ba9746f9b088affbf1a88cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.872799 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b7eae44ba9746f9b088affbf1a88cdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.873631 14069 raft_consensus.cc:482] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.873768 14069 raft_consensus.cc:504] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.874393 14069 leader_election.cc:288] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.874811 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8c01d6060ec04ecea2651ad7fc8b9436" } new_table_name: "default.Table_eea6723475c7480f86f5246eb849bf13" modify_external_catalogs: true
I20240506 19:56:48.875885 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcbeb22bfce436d931ff274248500fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.876156 12922 leader_election.cc:302] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:48.876889 13057 leader_election.cc:341] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.877061 13057 leader_election.cc:302] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:48.877440 14042 raft_consensus.cc:2726] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.877600 13249 ts_tablet_manager.cc:1398] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
W20240506 19:56:48.877835 12781 leader_election.cc:341] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:48.877657 14082 raft_consensus.cc:2781] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:48.878193 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 4d12edf17784486b9c598c2f29675060 (DEFAULT_TABLE table=default.Table_f9f5211355fa4a10b23a184e7c55ae7c [id=e311f1c887124c15a887b9def426858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.878268 13246 ts_tablet_manager.cc:1909] T 104337875517445f862d82ece3a84889 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:48.878597 12783 leader_election.cc:341] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:48.878608 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d (DEFAULT_TABLE table=default.Table_28fc76ba360b4dd4b805761b13f00f04 [id=f43d09fdf4ff4890a6fed18acf484153]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.878696 12783 leader_election.cc:302] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.878976 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.878930 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 4d12edf17784486b9c598c2f29675060 (DEFAULT_TABLE table=default.Table_f9f5211355fa4a10b23a184e7c55ae7c [id=e311f1c887124c15a887b9def426858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.879091 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 659) took 2479 ms. Trace:
I20240506 19:56:48.879096 14069 raft_consensus.cc:2726] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.879339 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d12edf17784486b9c598c2f29675060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.879900 13243 tablet_replica.cc:330] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:48.879992 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 574dbbe8a9a94cd7944b652a68b77403 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 574dbbe8a9a94cd7944b652a68b77403 already in progress: creating tablet
W20240506 19:56:48.880445 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7763959136c44674888916321bd6da24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7763959136c44674888916321bd6da24 already in progress: creating tablet
I20240506 19:56:48.879184 13246 rpcz_store.cc:276] 0506 19:56:46.399737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.400079 (+ 342us) service_pool.cc:225] Handling call
0506 19:56:48.828841 (+2428762us) ts_tablet_manager.cc:1120] Deleting tablet 104337875517445f862d82ece3a84889
0506 19:56:48.828850 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.856230 (+ 27380us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.866775 (+ 10545us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.879028 (+ 12253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428600}
I20240506 19:56:48.884265 13246 tablet_replica.cc:330] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.884382 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 104337875517445f862d82ece3a84889 (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]) successfully deleted
I20240506 19:56:48.884531 12819 tablet_service.cc:1467] Processing CreateTablet for tablet fbcbeb22bfce436d931ff274248500fd (DEFAULT_TABLE table=default.Table_ecc91044eebb4ad6b2799b1569836c8c [id=5f96354a891c42e6aed78d7cbb5d6f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.886595 13101 tablet_service.cc:1467] Processing CreateTablet for tablet fbcbeb22bfce436d931ff274248500fd (DEFAULT_TABLE table=default.Table_ecc91044eebb4ad6b2799b1569836c8c [id=5f96354a891c42e6aed78d7cbb5d6f00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.887065 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcbeb22bfce436d931ff274248500fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.887410 12809 tablet_service.cc:1467] Processing CreateTablet for tablet cf6d0f3900154554b7af9338c95b5122 (DEFAULT_TABLE table=default.Table_5e49e35a08cb438ebd5204ba40131503 [id=02204527b23b4c17b9edcdaa1391c439]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.887574 13094 tablet_service.cc:1467] Processing CreateTablet for tablet cf6d0f3900154554b7af9338c95b5122 (DEFAULT_TABLE table=default.Table_5e49e35a08cb438ebd5204ba40131503 [id=02204527b23b4c17b9edcdaa1391c439]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.884480 13246 raft_consensus.cc:2229] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.887903 13246 raft_consensus.cc:2258] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.888213 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6d0f3900154554b7af9338c95b5122. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.888491 13246 ts_tablet_manager.cc:1875] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.888603 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 979d789bc89a435b9e6cd8d6dc9c2037 (DEFAULT_TABLE table=default.Table_664bb720f8a04b2baf81c73ca7d3173d [id=3a92b0b6a5f149d0bec210c110fc6a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.889729 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 979d789bc89a435b9e6cd8d6dc9c2037 (DEFAULT_TABLE table=default.Table_664bb720f8a04b2baf81c73ca7d3173d [id=3a92b0b6a5f149d0bec210c110fc6a78]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.889730 12921 leader_election.cc:341] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:48.895388 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 979d789bc89a435b9e6cd8d6dc9c2037 (DEFAULT_TABLE table=default.Table_664bb720f8a04b2baf81c73ca7d3173d [id=3a92b0b6a5f149d0bec210c110fc6a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.895852 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979d789bc89a435b9e6cd8d6dc9c2037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.899008 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.899245 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d12edf17784486b9c598c2f29675060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.899442 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6d0f3900154554b7af9338c95b5122. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.899780 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979d789bc89a435b9e6cd8d6dc9c2037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.899962 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcbeb22bfce436d931ff274248500fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.902628 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de5e3db2bf34d6ab23b58d552d2f800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.902701 14082 raft_consensus.cc:686] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:48.918061 13246 ts_tablet_manager.cc:1888] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.918212 13246 log.cc:1196] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a8b3ed0b56aa478abb44347afca8fcf6
I20240506 19:56:48.918298 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 848b11a0d36b4cfb9e7769bf2b3aa8d2 (DEFAULT_TABLE table=default.Table_72a13ea44ea64858911b54c9e99739fc [id=cae4b61f8edf46ce85f5f517ced02305]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.918526 13246 ts_tablet_manager.cc:1909] T a8b3ed0b56aa478abb44347afca8fcf6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.918720 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848b11a0d36b4cfb9e7769bf2b3aa8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.919872 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 848b11a0d36b4cfb9e7769bf2b3aa8d2 (DEFAULT_TABLE table=default.Table_72a13ea44ea64858911b54c9e99739fc [id=cae4b61f8edf46ce85f5f517ced02305]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.919497 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 660) took 2510 ms. Trace:
I20240506 19:56:48.920614 13246 rpcz_store.cc:276] 0506 19:56:46.409445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.409505 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:48.884170 (+2474665us) ts_tablet_manager.cc:1120] Deleting tablet a8b3ed0b56aa478abb44347afca8fcf6
0506 19:56:48.884181 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.906938 (+ 22757us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.917929 (+ 10991us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.919450 (+ 1521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474479}
I20240506 19:56:48.920724 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 4d09e7b64b6d45618c6595c1013922df (DEFAULT_TABLE table=default.Table_912ad4a3913f41e394764648805c3a30 [id=01881436327441a384925f571f99bf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.921576 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d (DEFAULT_TABLE table=default.Table_a2925684b6564128a1301dfe07064f5f [id=5bc54d89fcab44a186cdaaa0656b144f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.921675 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 661) took 2511 ms. Trace:
I20240506 19:56:48.921895 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.921841 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 371d450876ff40a59b859f72f4e20c5a (DEFAULT_TABLE table=default.Table_6f20f6b912c249479dbf461093583817 [id=d401826620ea46ad8df6016c1f1cf968]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.924151 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d (DEFAULT_TABLE table=default.Table_a2925684b6564128a1301dfe07064f5f [id=5bc54d89fcab44a186cdaaa0656b144f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.925560 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 4d09e7b64b6d45618c6595c1013922df (DEFAULT_TABLE table=default.Table_912ad4a3913f41e394764648805c3a30 [id=01881436327441a384925f571f99bf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.921790 13246 rpcz_store.cc:276] 0506 19:56:46.410355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.410405 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:48.920871 (+2510466us) ts_tablet_manager.cc:1120] Deleting tablet 449eeb67b602425bb40caa69b6a78a5a
0506 19:56:48.921503 (+ 632us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.921638 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2510296}
I20240506 19:56:48.925875 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d09e7b64b6d45618c6595c1013922df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.925882 13246 tablet_replica.cc:330] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:48.926102 13246 raft_consensus.cc:2229] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.926282 13246 raft_consensus.cc:2258] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:48.926683 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 449eeb67b602425bb40caa69b6a78a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 449eeb67b602425bb40caa69b6a78a5a already in progress: creating tablet
I20240506 19:56:48.926856 13246 ts_tablet_manager.cc:1875] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.927482 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848b11a0d36b4cfb9e7769bf2b3aa8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.928282 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371d450876ff40a59b859f72f4e20c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.929040 13242 ts_tablet_manager.cc:1888] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.929133 13242 log.cc:1196] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b5a443c0d35741b9966b32a7808a97e2
I20240506 19:56:48.929428 13242 ts_tablet_manager.cc:1909] T b5a443c0d35741b9966b32a7808a97e2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:48.929585 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 371d450876ff40a59b859f72f4e20c5a (DEFAULT_TABLE table=default.Table_6f20f6b912c249479dbf461093583817 [id=d401826620ea46ad8df6016c1f1cf968]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.929704 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 563) took 4478 ms. Trace:
I20240506 19:56:48.929900 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371d450876ff40a59b859f72f4e20c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.931950 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d09e7b64b6d45618c6595c1013922df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.933584 12817 tablet_service.cc:1467] Processing CreateTablet for tablet cf6628f9d67f43b7a5be2913d9446d62 (DEFAULT_TABLE table=default.Table_76d78f0c434c45508bfd15806c510293 [id=ada2217dde264b8dabf49f9a8bcb25c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.933904 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6628f9d67f43b7a5be2913d9446d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.939605 13245 tablet_bootstrap.cc:492] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:48.939807 13245 ts_tablet_manager.cc:1367] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.007s
I20240506 19:56:48.940528 13245 raft_consensus.cc:348] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.940727 13245 raft_consensus.cc:374] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:48.940790 13245 raft_consensus.cc:729] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:48.940984 13245 consensus_queue.cc:260] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.941507 13245 ts_tablet_manager.cc:1398] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:48.941689 14069 raft_consensus.cc:482] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.941783 13245 tablet_bootstrap.cc:492] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:48.941815 14069 raft_consensus.cc:504] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.942377 14069 leader_election.cc:288] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.943229 13245 tablet_bootstrap.cc:654] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.944721 12819 tablet_service.cc:1467] Processing CreateTablet for tablet c264c66cfec443c7a38677237ae2ba31 (DEFAULT_TABLE table=default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:48.945060 12783 leader_election.cc:341] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 00de09ddcdd948a6a3500a6c9d814c14
I20240506 19:56:48.929838 13242 rpcz_store.cc:276] 0506 19:56:44.451446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.451874 (+ 428us) service_pool.cc:225] Handling call
0506 19:56:48.771850 (+4319976us) ts_tablet_manager.cc:1120] Deleting tablet b5a443c0d35741b9966b32a7808a97e2
0506 19:56:48.771859 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.781230 (+ 9371us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.928934 (+147704us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.929685 (+ 751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4319787}
I20240506 19:56:48.946071 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b5a443c0d35741b9966b32a7808a97e2 (table default.Table_1a5485067c1a4e529059f5434babbdc7 [id=54286945a7554349b48aa6511f84326c]) successfully deleted
I20240506 19:56:48.946172 13246 ts_tablet_manager.cc:1888] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:48.946276 13246 log.cc:1196] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/da41ef03182445f1946584c2bf015139
I20240506 19:56:48.946574 13246 ts_tablet_manager.cc:1909] T da41ef03182445f1946584c2bf015139 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:48.946642 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c264c66cfec443c7a38677237ae2ba31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.946993 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 564) took 4493 ms. Trace:
I20240506 19:56:48.947077 13242 rpcz_store.cc:276] 0506 19:56:44.453703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.453807 (+ 104us) service_pool.cc:225] Handling call
0506 19:56:48.946875 (+4493068us) ts_tablet_manager.cc:1120] Deleting tablet 0aa8f9aaca444c839a88c07f58cb6ad6
0506 19:56:48.946885 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.946976 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492874}
I20240506 19:56:48.947333 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 665) took 2508 ms. Trace:
I20240506 19:56:48.947399 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 565) took 4493 ms. Trace:
I20240506 19:56:48.947465 13242 rpcz_store.cc:276] 0506 19:56:44.454365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.454413 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:48.947298 (+4492885us) ts_tablet_manager.cc:1120] Deleting tablet 30ad527c3f2a4b53affa928e149db1df
0506 19:56:48.947308 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.947384 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492648}
I20240506 19:56:48.947417 13246 rpcz_store.cc:276] 0506 19:56:46.439156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.439223 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:48.925819 (+2486596us) ts_tablet_manager.cc:1120] Deleting tablet da41ef03182445f1946584c2bf015139
0506 19:56:48.925828 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.936696 (+ 10868us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.946051 (+ 9355us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:48.947314 (+ 1263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2486300}
I20240506 19:56:48.947717 13242 tablet_replica.cc:330] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:48.947806 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 677) took 2210 ms. Trace:
W20240506 19:56:48.947839 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aa8f9aaca444c839a88c07f58cb6ad6 already in progress: creating tablet
I20240506 19:56:48.947899 13242 raft_consensus.cc:2229] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.947872 13246 rpcz_store.cc:276] 0506 19:56:46.737517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.737572 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:48.947686 (+2210114us) ts_tablet_manager.cc:1120] Deleting tablet 9227f74e85914cafabad913dc6dfba04
0506 19:56:48.947694 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.947791 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2209854}
I20240506 19:56:48.948048 13242 raft_consensus.cc:2258] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.948180 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 676) took 2210 ms. Trace:
I20240506 19:56:48.948249 13246 rpcz_store.cc:276] 0506 19:56:46.737377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.738836 (+ 1459us) service_pool.cc:225] Handling call
0506 19:56:48.948074 (+2209238us) ts_tablet_manager.cc:1120] Deleting tablet 9fed2f12eda54be0879b1e9c33ca3e31
0506 19:56:48.948081 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.948160 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2209004}
W20240506 19:56:48.948377 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 30ad527c3f2a4b53affa928e149db1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ad527c3f2a4b53affa928e149db1df already in progress: creating tablet
I20240506 19:56:48.948519 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 678) took 2137 ms. Trace:
I20240506 19:56:48.948562 13242 ts_tablet_manager.cc:1875] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.948594 13246 rpcz_store.cc:276] 0506 19:56:46.811471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.811519 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:48.948429 (+2136910us) ts_tablet_manager.cc:1120] Deleting tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa
0506 19:56:48.948437 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.948506 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136707}
I20240506 19:56:48.948866 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 679) took 1797 ms. Trace:
I20240506 19:56:48.948931 13246 rpcz_store.cc:276] 0506 19:56:47.151153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.151217 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:48.948779 (+1797562us) ts_tablet_manager.cc:1120] Deleting tablet 01c32eba2031446ab419cac64068068e
0506 19:56:48.948787 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.948853 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792710}
I20240506 19:56:48.949191 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 708) took 1631 ms. Trace:
I20240506 19:56:48.950271 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ef6c459b3eed45d08d709ab6f797aa4a" } new_table_name: "default.Table_643dbcd828ba4989ba329328a3708ae3" modify_external_catalogs: true
I20240506 19:56:48.951694 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_96341728d3094446990de463a07dafd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:48.952241 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96341728d3094446990de463a07dafd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:48.953711 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:48.953915 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 67e8b8be953a4a5784f2d098fd6364e3 (table default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb [id=7a2c513cd8e74817b4a391c9c7932085]). Expected version 1 got 0
I20240506 19:56:48.954103 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]). Expected version 2 got 0
I20240506 19:56:48.954239 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 63f56d2180b1472a8168fb3a63589065 (table default.Table_ab69ac48aa7a4934a9f383276277cf10 [id=493f88f528114491b898c4a1156712fc]). Expected version 2 got 0
I20240506 19:56:48.949251 13246 rpcz_store.cc:276] 0506 19:56:47.317887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.576090 (+258203us) service_pool.cc:225] Handling call
0506 19:56:48.949104 (+1373014us) ts_tablet_manager.cc:1120] Deleting tablet 1d53f3daaa2949a2b6883435f031dc37
0506 19:56:48.949111 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.949177 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372834}
I20240506 19:56:48.958470 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979d789bc89a435b9e6cd8d6dc9c2037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.958722 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 371d450876ff40a59b859f72f4e20c5a (DEFAULT_TABLE table=default.Table_6f20f6b912c249479dbf461093583817 [id=d401826620ea46ad8df6016c1f1cf968]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.959137 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371d450876ff40a59b859f72f4e20c5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:48.961282 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4973e600038742a6bec0bb2d8f3551fe consensus state has no leader
I20240506 19:56:48.961905 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 4d09e7b64b6d45618c6595c1013922df (DEFAULT_TABLE table=default.Table_912ad4a3913f41e394764648805c3a30 [id=01881436327441a384925f571f99bf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.962275 14069 raft_consensus.cc:482] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.961503 14082 consensus_queue.cc:237] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.962420 14069 raft_consensus.cc:504] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:48.962961 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet da41ef03182445f1946584c2bf015139 (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
I20240506 19:56:48.963028 14069 leader_election.cc:288] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.963176 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 709) took 1638 ms. Trace:
W20240506 19:56:48.963225 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9227f74e85914cafabad913dc6dfba04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9227f74e85914cafabad913dc6dfba04 already in progress: creating tablet
I20240506 19:56:48.963215 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 848b11a0d36b4cfb9e7769bf2b3aa8d2 (DEFAULT_TABLE table=default.Table_72a13ea44ea64858911b54c9e99739fc [id=cae4b61f8edf46ce85f5f517ced02305]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.963268 13246 rpcz_store.cc:276] 0506 19:56:47.324604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.576287 (+251683us) service_pool.cc:225] Handling call
0506 19:56:48.957840 (+1381553us) ts_tablet_manager.cc:1120] Deleting tablet 60d1f01c0edb4f6b8a621c1643ca1afb
0506 19:56:48.957850 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:48.963153 (+ 5303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381391}
W20240506 19:56:48.963435 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9fed2f12eda54be0879b1e9c33ca3e31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fed2f12eda54be0879b1e9c33ca3e31 already in progress: creating tablet
W20240506 19:56:48.963621 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa already in progress: creating tablet
I20240506 19:56:48.964166 12954 tablet_service.cc:1467] Processing CreateTablet for tablet cf6628f9d67f43b7a5be2913d9446d62 (DEFAULT_TABLE table=default.Table_76d78f0c434c45508bfd15806c510293 [id=ada2217dde264b8dabf49f9a8bcb25c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.964447 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9ecd2e034e49e49772db7435da3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:48.964486 12966 tablet_service.cc:1467] Processing CreateTablet for tablet cf6d0f3900154554b7af9338c95b5122 (DEFAULT_TABLE table=default.Table_5e49e35a08cb438ebd5204ba40131503 [id=02204527b23b4c17b9edcdaa1391c439]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.964697 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_af16118ff86a4673a5e238050eb735de" } modify_external_catalogs: true
I20240506 19:56:48.964866 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_af16118ff86a4673a5e238050eb735de" } modify_external_catalogs: true
I20240506 19:56:48.968026 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:48.968593 13093 tablet_service.cc:1467] Processing CreateTablet for tablet cf6628f9d67f43b7a5be2913d9446d62 (DEFAULT_TABLE table=default.Table_76d78f0c434c45508bfd15806c510293 [id=ada2217dde264b8dabf49f9a8bcb25c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.969036 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6628f9d67f43b7a5be2913d9446d62. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:48.969054 12781 leader_election.cc:341] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 00de09ddcdd948a6a3500a6c9d814c14
I20240506 19:56:48.969197 12781 leader_election.cc:302] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.969327 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d09e7b64b6d45618c6595c1013922df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.969579 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848b11a0d36b4cfb9e7769bf2b3aa8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.969622 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d (DEFAULT_TABLE table=default.Table_a2925684b6564128a1301dfe07064f5f [id=5bc54d89fcab44a186cdaaa0656b144f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.969834 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6d0f3900154554b7af9338c95b5122. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.970039 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.973397 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6628f9d67f43b7a5be2913d9446d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.973448 14069 raft_consensus.cc:2726] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.973744 14080 raft_consensus.cc:482] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:48.973888 14080 raft_consensus.cc:504] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:48.974628 14080 leader_election.cc:288] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:48.975399 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6a2c98fd374b8cbc5a65069bd4cb5b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:48.976145 12921 leader_election.cc:341] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:48.976567 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1d53f3daaa2949a2b6883435f031dc37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d53f3daaa2949a2b6883435f031dc37 already in progress: creating tablet
W20240506 19:56:48.976786 12781 leader_election.cc:341] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:48.978458 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6a2c98fd374b8cbc5a65069bd4cb5b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:48.978683 13120 raft_consensus.cc:2454] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:48.979001 12783 leader_election.cc:341] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:48.979110 12783 leader_election.cc:302] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:48.979552 14069 raft_consensus.cc:2726] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:48.980314 12922 leader_election.cc:302] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:48.980818 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 60d1f01c0edb4f6b8a621c1643ca1afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d1f01c0edb4f6b8a621c1643ca1afb already in progress: creating tablet
I20240506 19:56:48.980892 14080 raft_consensus.cc:2781] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:48.981168 14080 raft_consensus.cc:482] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:48.981180 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.981254 14080 raft_consensus.cc:3037] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:48.981628 13249 tablet_bootstrap.cc:492] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:48.983465 13249 tablet_bootstrap.cc:654] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:48.983567 13243 raft_consensus.cc:2229] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:48.983783 13243 raft_consensus.cc:2258] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:48.984524 13243 ts_tablet_manager.cc:1875] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:48.986943 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_54e2ae0fb2324776911994a064b540f0" } modify_external_catalogs: true
I20240506 19:56:48.988762 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 8689ec0a8e134bceb010053b1f5a343d (DEFAULT_TABLE table=default.Table_da4848522d5447739671166038af13aa [id=1a8a60e33c824ddd988264116d1665c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.989004 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 8689ec0a8e134bceb010053b1f5a343d (DEFAULT_TABLE table=default.Table_da4848522d5447739671166038af13aa [id=1a8a60e33c824ddd988264116d1665c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.989169 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8689ec0a8e134bceb010053b1f5a343d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.989517 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8689ec0a8e134bceb010053b1f5a343d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:48.991475 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_54e2ae0fb2324776911994a064b540f0" } modify_external_catalogs: true
I20240506 19:56:48.992540 13243 ts_tablet_manager.cc:1888] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:48.992643 13243 log.cc:1196] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/796628a0859c4511a8cb35e495dc8886
I20240506 19:56:48.993400 12952 tablet_service.cc:1467] Processing CreateTablet for tablet c264c66cfec443c7a38677237ae2ba31 (DEFAULT_TABLE table=default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:48.993832 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c264c66cfec443c7a38677237ae2ba31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.000968 13244 tablet_bootstrap.cc:492] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.001128 13244 ts_tablet_manager.cc:1367] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.178s user 0.001s sys 0.013s
I20240506 19:56:49.001591 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 37353519fb1343349193ea6c82ecda89 (DEFAULT_TABLE table=default.Table_8c01d6060ec04ecea2651ad7fc8b9436 [id=62405a21681d4effaea8f463eb15bf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.001713 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 37353519fb1343349193ea6c82ecda89 (DEFAULT_TABLE table=default.Table_8c01d6060ec04ecea2651ad7fc8b9436 [id=62405a21681d4effaea8f463eb15bf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.001996 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37353519fb1343349193ea6c82ecda89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.002085 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37353519fb1343349193ea6c82ecda89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.001948 13244 raft_consensus.cc:348] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.002153 13244 raft_consensus.cc:374] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.002208 13244 raft_consensus.cc:729] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.002411 13244 consensus_queue.cc:260] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.002916 13244 ts_tablet_manager.cc:1398] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:49.003170 13244 tablet_bootstrap.cc:492] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.005355 13244 tablet_bootstrap.cc:654] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.007843 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7aa1324e9749948c311f5c291ff06d (DEFAULT_TABLE table=default.Table_13e6e7d68e57433ebed02111652c43a9 [id=fe195172a40d45a5aef9388abdc14672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.008237 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7aa1324e9749948c311f5c291ff06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.008728 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7aa1324e9749948c311f5c291ff06d (DEFAULT_TABLE table=default.Table_13e6e7d68e57433ebed02111652c43a9 [id=fe195172a40d45a5aef9388abdc14672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.012212 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 710) took 1687 ms. Trace:
I20240506 19:56:49.012312 13246 rpcz_store.cc:276] 0506 19:56:47.324718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.576457 (+251739us) service_pool.cc:225] Handling call
0506 19:56:49.012065 (+1435608us) ts_tablet_manager.cc:1120] Deleting tablet 2c96f014293242be9640515697c0733b
0506 19:56:49.012073 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.012185 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386957}
I20240506 19:56:49.012945 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 711) took 1686 ms. Trace:
I20240506 19:56:49.013017 13246 rpcz_store.cc:276] 0506 19:56:47.326237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.576586 (+250349us) service_pool.cc:225] Handling call
0506 19:56:49.012835 (+1436249us) ts_tablet_manager.cc:1120] Deleting tablet 1b89fb24d94042e1bfa772265a991680
0506 19:56:49.012845 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.012929 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436131}
I20240506 19:56:49.013770 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 712) took 1547 ms. Trace:
I20240506 19:56:49.013836 13246 rpcz_store.cc:276] 0506 19:56:47.465872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.576709 (+110837us) service_pool.cc:225] Handling call
0506 19:56:49.013665 (+1436956us) ts_tablet_manager.cc:1120] Deleting tablet f759b560e2ab41a6a076976ccca811d1
0506 19:56:49.013674 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.013756 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436845}
I20240506 19:56:49.014933 13249 tablet_bootstrap.cc:492] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.014964 13245 tablet_bootstrap.cc:492] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.015080 13249 ts_tablet_manager.cc:1367] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.000s
I20240506 19:56:49.015429 13245 ts_tablet_manager.cc:1367] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.005s
I20240506 19:56:49.016160 13245 raft_consensus.cc:348] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.015847 13249 raft_consensus.cc:348] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.016348 13245 raft_consensus.cc:374] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.016407 13245 raft_consensus.cc:729] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.016422 13249 raft_consensus.cc:374] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.016479 13249 raft_consensus.cc:729] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.016729 13245 consensus_queue.cc:260] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.016793 13249 consensus_queue.cc:260] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.017251 13245 ts_tablet_manager.cc:1398] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:49.017551 13245 tablet_bootstrap.cc:492] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.020612 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 37353519fb1343349193ea6c82ecda89 (DEFAULT_TABLE table=default.Table_8c01d6060ec04ecea2651ad7fc8b9436 [id=62405a21681d4effaea8f463eb15bf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.020606 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dccf1c1b22014892a25a3ea6da1ecf29" } modify_external_catalogs: true
I20240506 19:56:49.020867 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dccf1c1b22014892a25a3ea6da1ecf29" } modify_external_catalogs: true
I20240506 19:56:49.020964 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_61659b4f08a1437f953f0e5345dcfc8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.021404 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7aa1324e9749948c311f5c291ff06d (DEFAULT_TABLE table=default.Table_13e6e7d68e57433ebed02111652c43a9 [id=fe195172a40d45a5aef9388abdc14672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.021701 14075 raft_consensus.cc:482] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.021832 14075 raft_consensus.cc:504] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.022464 14075 leader_election.cc:288] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.022445 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_41ecc96260e940ca86684d10d102665d" } modify_external_catalogs: true
W20240506 19:56:49.022677 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61659b4f08a1437f953f0e5345dcfc8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.022832 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_41ecc96260e940ca86684d10d102665d" } modify_external_catalogs: true
W20240506 19:56:49.023608 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2c96f014293242be9640515697c0733b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c96f014293242be9640515697c0733b already in progress: creating tablet
I20240506 19:56:49.023677 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_95c37af6ef334405b443163c4eb60668"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.023917 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f759b560e2ab41a6a076976ccca811d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f759b560e2ab41a6a076976ccca811d1 already in progress: creating tablet
I20240506 19:56:49.024657 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378759ca310f453fba45d48879e2dd1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.025538 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 8689ec0a8e134bceb010053b1f5a343d (DEFAULT_TABLE table=default.Table_da4848522d5447739671166038af13aa [id=1a8a60e33c824ddd988264116d1665c5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:49.025794 12921 leader_election.cc:341] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.026255 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378759ca310f453fba45d48879e2dd1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:49.026463 13120 raft_consensus.cc:2454] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:49.026511 13096 tablet_service.cc:1467] Processing CreateTablet for tablet c264c66cfec443c7a38677237ae2ba31 (DEFAULT_TABLE table=default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.026906 12922 leader_election.cc:302] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.027392 14075 raft_consensus.cc:2781] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:49.027468 14075 raft_consensus.cc:482] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:49.027529 14075 raft_consensus.cc:3037] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.021836 13249 ts_tablet_manager.cc:1398] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240506 19:56:49.027873 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95c37af6ef334405b443163c4eb60668 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.028671 13249 tablet_bootstrap.cc:492] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.029243 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7aa1324e9749948c311f5c291ff06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.032939 13249 tablet_bootstrap.cc:654] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.035868 14080 raft_consensus.cc:504] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.037021 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6a2c98fd374b8cbc5a65069bd4cb5b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:49.037688 14080 leader_election.cc:288] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:49.037945 12921 leader_election.cc:341] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.038590 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c6a2c98fd374b8cbc5a65069bd4cb5b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:49.038791 13120 raft_consensus.cc:3037] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.044047 13120 raft_consensus.cc:2454] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:49.044771 12922 leader_election.cc:302] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.045513 14080 raft_consensus.cc:2781] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.048538 13244 tablet_bootstrap.cc:492] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.048835 13244 ts_tablet_manager.cc:1367] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.007s
I20240506 19:56:49.050523 13244 raft_consensus.cc:348] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.051120 13244 raft_consensus.cc:374] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.051271 13244 raft_consensus.cc:729] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.051839 13244 consensus_queue.cc:260] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.053263 13244 ts_tablet_manager.cc:1398] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240506 19:56:49.053916 13244 tablet_bootstrap.cc:492] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.046641 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 723) took 1075 ms. Trace:
I20240506 19:56:49.055152 13246 rpcz_store.cc:276] 0506 19:56:47.971299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.971938 (+ 639us) service_pool.cc:225] Handling call
0506 19:56:49.045697 (+1073759us) ts_tablet_manager.cc:1120] Deleting tablet 2a8e331dc24e4d06bf81c8a4b7d70156
0506 19:56:49.045710 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.046431 (+ 721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059815}
I20240506 19:56:49.058171 13095 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7f7171d06948388bc0fdb3514202ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.057659 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 727) took 1006 ms. Trace:
W20240506 19:56:49.059039 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a8e331dc24e4d06bf81c8a4b7d70156 already in progress: creating tablet
I20240506 19:56:49.059281 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37353519fb1343349193ea6c82ecda89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.059681 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7aa1324e9749948c311f5c291ff06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.060014 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8689ec0a8e134bceb010053b1f5a343d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.060537 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c264c66cfec443c7a38677237ae2ba31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.058279 13246 rpcz_store.cc:276] 0506 19:56:48.051129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.051181 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:49.056856 (+1005675us) ts_tablet_manager.cc:1120] Deleting tablet df04fb90230d4bf4a060db993adafd14
0506 19:56:49.056976 (+ 120us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.057521 (+ 545us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005335}
I20240506 19:56:49.078948 14042 raft_consensus.cc:482] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.079118 14042 raft_consensus.cc:504] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.079773 14042 leader_election.cc:288] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.080102 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bafd3c4706f74e7da36a8f3538760f43" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:49.080855 13058 leader_election.cc:341] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.081106 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d9668767a7184407ac5f03e85864411e" } new_table_name: "default.Table_3f262965cba54133b5872f383dd0c32c" modify_external_catalogs: true
I20240506 19:56:49.081526 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 08535fa1600b41ff919fefa93cfc4c78
I20240506 19:56:49.082247 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ab69ac48aa7a4934a9f383276277cf10" } modify_external_catalogs: true
I20240506 19:56:49.082386 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 08535fa1600b41ff919fefa93cfc4c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.082438 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ab69ac48aa7a4934a9f383276277cf10" } modify_external_catalogs: true
I20240506 19:56:49.083804 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d6edff8e7e6945aa926aa1486cbbc2bc
I20240506 19:56:49.084466 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet d6edff8e7e6945aa926aa1486cbbc2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.084537 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5ac61897169e4a91bbe63d32fce65440" } new_table_name: "default.Table_23fc1ee2934c454982e8fd45aad54dd1" modify_external_catalogs: true
I20240506 19:56:49.086077 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 08535fa1600b41ff919fefa93cfc4c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:49.087725 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37353519fb1343349193ea6c82ecda89 consensus state has no leader
W20240506 19:56:49.088115 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f476f745e0784f7f9886484367dd8c22 consensus state has no leader
I20240506 19:56:49.089624 13246 tablet_replica.cc:330] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.089826 13246 raft_consensus.cc:2229] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.089953 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet fdef0a40f2e346a798b01e83a960de09
I20240506 19:56:49.090003 13246 raft_consensus.cc:2258] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.090770 13246 ts_tablet_manager.cc:1875] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.092142 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet d6edff8e7e6945aa926aa1486cbbc2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.092314 13243 ts_tablet_manager.cc:1909] T 796628a0859c4511a8cb35e495dc8886 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:49.092337 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 08535fa1600b41ff919fefa93cfc4c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.093094 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e7329b22414da886dee4e1ec6bf3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.094220 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet fdef0a40f2e346a798b01e83a960de09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.095150 13245 tablet_bootstrap.cc:654] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.099370 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet fdef0a40f2e346a798b01e83a960de09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.099555 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet d6edff8e7e6945aa926aa1486cbbc2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:49.100860 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet df04fb90230d4bf4a060db993adafd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df04fb90230d4bf4a060db993adafd14 already in progress: creating tablet
I20240506 19:56:49.101629 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 08535fa1600b41ff919fefa93cfc4c78 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:49.102125 14075 raft_consensus.cc:504] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.103392 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378759ca310f453fba45d48879e2dd1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:49.103998 12921 leader_election.cc:341] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.104590 13244 tablet_bootstrap.cc:654] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.108640 14042 raft_consensus.cc:482] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.108788 14042 raft_consensus.cc:504] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.109455 14042 leader_election.cc:288] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.110145 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.110677 13245 tablet_bootstrap.cc:492] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.110821 13245 ts_tablet_manager.cc:1367] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.001s
I20240506 19:56:49.113819 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]) successfully deleted
W20240506 19:56:49.114049 13058 leader_election.cc:341] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.114289 13243 tablet_replica.cc:330] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:49.114491 13243 raft_consensus.cc:2229] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:49.114725 13243 pending_rounds.cc:62] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:49.114727 13244 tablet_bootstrap.cc:492] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.114850 13244 ts_tablet_manager.cc:1367] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20240506 19:56:49.115633 13244 raft_consensus.cc:348] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.115821 13244 raft_consensus.cc:374] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.115878 13244 raft_consensus.cc:729] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.116103 13244 consensus_queue.cc:260] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.116642 13244 ts_tablet_manager.cc:1398] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:49.116919 13244 tablet_bootstrap.cc:492] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.114773 13243 pending_rounds.cc:68] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744073694330880 op_type: NO_OP noop_request { }
I20240506 19:56:49.117168 13243 raft_consensus.cc:2866] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:49.117214 14075 leader_election.cc:288] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.117570 13243 raft_consensus.cc:2258] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.122380 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 63f56d2180b1472a8168fb3a63589065
I20240506 19:56:49.122928 13243 ts_tablet_manager.cc:1875] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:49.123255 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c2eb5dac9d34fb09119868bf5a85803 consensus state has no leader
I20240506 19:56:49.124286 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 79592a4ecacb4324b2278e1b9a75496c
W20240506 19:56:49.125823 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 85365758f6a64536ab61e8349de63dea (table default.Table_294ef89cfe854b01a260a32748b84c85 [id=c266c557aaf24c0d9eaba0cc04b6988f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 85365758f6a64536ab61e8349de63dea consensus state has no leader
I20240506 19:56:49.125808 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 79592a4ecacb4324b2278e1b9a75496c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.125747 13245 raft_consensus.cc:348] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.125952 13245 raft_consensus.cc:374] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.126004 13245 raft_consensus.cc:729] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.126113 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f56d2180b1472a8168fb3a63589065 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.126703 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f56d2180b1472a8168fb3a63589065 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.126868 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f56d2180b1472a8168fb3a63589065 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.126899 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 79592a4ecacb4324b2278e1b9a75496c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.130462 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 79592a4ecacb4324b2278e1b9a75496c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.126191 13245 consensus_queue.cc:260] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.137923 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet b899fb38bfed4e02b3708f6abec80e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.138669 13245 ts_tablet_manager.cc:1398] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240506 19:56:49.141161 13243 ts_tablet_manager.cc:1888] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:49.142087 13245 tablet_bootstrap.cc:492] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.148131 13243 log.cc:1196] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/30ad527c3f2a4b53affa928e149db1df
I20240506 19:56:49.148609 13243 ts_tablet_manager.cc:1909] T 30ad527c3f2a4b53affa928e149db1df P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:49.150318 13245 tablet_bootstrap.cc:654] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.150995 13246 ts_tablet_manager.cc:1888] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.152887 13246 log.cc:1196] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:49.153574 13246 ts_tablet_manager.cc:1909] T 7d23f0a5ff6d430282256e9bddc186b9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.162479 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_50e723eab6834893b76afae99961fb99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.162689 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_71a4779621224f0aa1c1b137ad0de871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.163385 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71a4779621224f0aa1c1b137ad0de871 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.163237 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_807ff2b24da64f7396c8dc15bdc46c88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.163941 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_807ff2b24da64f7396c8dc15bdc46c88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:49.165088 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50e723eab6834893b76afae99961fb99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.141718 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 0114edac976844619e438c17bb959010 with delete_type TABLET_DATA_DELETED (replaced by tablet 6352d2c630204df3a24e299e167f92d4) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.144441 13244 tablet_bootstrap.cc:654] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.166592 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 729) took 1105 ms. Trace:
I20240506 19:56:49.168792 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.169533 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.171087 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_27b96f0770c845b38361dfc9ac3f86f1" } new_table_name: "default.Table_9eea3fa543d74708aed9bf6f5e378d89" modify_external_catalogs: true
I20240506 19:56:49.146387 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a8b12f36e74c95ad6eaf30e432f3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.171873 13244 tablet_bootstrap.cc:492] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.172037 13244 ts_tablet_manager.cc:1367] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.002s
I20240506 19:56:49.172863 13244 raft_consensus.cc:348] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.173087 13244 raft_consensus.cc:374] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.172958 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aa2cfdcd14e74072be6f657fcb05127e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.173159 13244 raft_consensus.cc:729] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.173528 13244 consensus_queue.cc:260] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:49.173745 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa2cfdcd14e74072be6f657fcb05127e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.175406 13244 ts_tablet_manager.cc:1398] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:49.175621 14069 raft_consensus.cc:482] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.175743 14069 raft_consensus.cc:504] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.176340 14069 leader_election.cc:288] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.176546 13244 tablet_bootstrap.cc:492] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:49.179360 12783 leader_election.cc:341] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:49.185151 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet d05c5ca7ff864fa98f3494032729e978 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a7c61244620487bbbac26dcdbf8a38c) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.186424 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ff37f251dcdb4e1d96089c80f2ed4e82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.187197 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.188443 13242 ts_tablet_manager.cc:1888] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.188555 13242 log.cc:1196] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2751074a54ed4bcea6979e6ad11c2efe
I20240506 19:56:49.188858 13242 ts_tablet_manager.cc:1909] T 2751074a54ed4bcea6979e6ad11c2efe P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.189191 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 571) took 4440 ms. Trace:
I20240506 19:56:49.189307 13242 rpcz_store.cc:276] 0506 19:56:44.748385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.748491 (+ 106us) service_pool.cc:225] Handling call
0506 19:56:48.947674 (+4199183us) ts_tablet_manager.cc:1120] Deleting tablet 2751074a54ed4bcea6979e6ad11c2efe
0506 19:56:48.947684 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.031538 (+ 83854us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.188322 (+156784us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.189161 (+ 839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4198740}
I20240506 19:56:49.190124 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2751074a54ed4bcea6979e6ad11c2efe (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]) successfully deleted
I20240506 19:56:49.190737 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 572) took 4420 ms. Trace:
I20240506 19:56:49.190893 13242 rpcz_store.cc:276] 0506 19:56:44.769800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.769994 (+ 194us) service_pool.cc:225] Handling call
0506 19:56:49.190570 (+4420576us) ts_tablet_manager.cc:1120] Deleting tablet 7de7397db1e74b4488a3f9a70136bccb
0506 19:56:49.190584 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.190711 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4420240}
W20240506 19:56:49.191555 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7de7397db1e74b4488a3f9a70136bccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7de7397db1e74b4488a3f9a70136bccb already in progress: creating tablet
I20240506 19:56:49.192077 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 579) took 4285 ms. Trace:
I20240506 19:56:49.192318 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet fdef0a40f2e346a798b01e83a960de09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.166736 13246 rpcz_store.cc:276] 0506 19:56:48.061150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.061746 (+ 596us) service_pool.cc:225] Handling call
0506 19:56:49.089570 (+1027824us) ts_tablet_manager.cc:1120] Deleting tablet 7d23f0a5ff6d430282256e9bddc186b9
0506 19:56:49.089582 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.122071 (+ 32489us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.150859 (+ 28788us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.166555 (+ 15696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027578}
I20240506 19:56:49.194240 13246 tablet_replica.cc:330] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.194993 13246 raft_consensus.cc:2229] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.195215 13246 raft_consensus.cc:2258] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.195861 13246 ts_tablet_manager.cc:1875] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.195930 13244 tablet_bootstrap.cc:654] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.198628 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 7d23f0a5ff6d430282256e9bddc186b9 (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:49.201087 14042 raft_consensus.cc:482] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.201244 14042 raft_consensus.cc:504] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.201566 13243 tablet_replica.cc:330] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:49.201772 13243 raft_consensus.cc:2229] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:49.202020 13243 pending_rounds.cc:62] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:49.201996 14042 leader_election.cc:288] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.202065 13243 pending_rounds.cc:68] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744074142195712 op_type: NO_OP noop_request { }
I20240506 19:56:49.202183 13243 raft_consensus.cc:2866] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:49.202270 13243 raft_consensus.cc:2258] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.202901 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e8b8be953a4a5784f2d098fd6364e3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:49.203462 13058 leader_election.cc:341] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.215924 13246 ts_tablet_manager.cc:1888] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.216089 13246 log.cc:1196] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:49.216387 13246 ts_tablet_manager.cc:1909] T 9e3d13322b4a4a2db3b6d425679f68ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.217284 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 730) took 1156 ms. Trace:
I20240506 19:56:49.217425 13246 rpcz_store.cc:276] 0506 19:56:48.061217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.062044 (+ 827us) service_pool.cc:225] Handling call
0506 19:56:49.194183 (+1132139us) ts_tablet_manager.cc:1120] Deleting tablet 9e3d13322b4a4a2db3b6d425679f68ed
0506 19:56:49.194196 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.210345 (+ 16149us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.215803 (+ 5458us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.217252 (+ 1449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131987}
I20240506 19:56:49.217832 13246 tablet_replica.cc:330] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.218091 13246 raft_consensus.cc:2229] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.218268 13246 raft_consensus.cc:2258] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.218892 13246 ts_tablet_manager.cc:1875] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.222406 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 378759ca310f453fba45d48879e2dd1a with delete_type TABLET_DATA_DELETED (replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.222597 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.222879 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 30ad527c3f2a4b53affa928e149db1df (table default.Table_052f7d1eab54464d8743bd859c9f5bad [id=cdb5ce64abc74bb3bf60d7aa727c515f]) successfully deleted
W20240506 19:56:49.223263 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8689ec0a8e134bceb010053b1f5a343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8689ec0a8e134bceb010053b1f5a343d already in progress: creating tablet
W20240506 19:56:49.223610 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9e17d0a261734d45866525b35de7cba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e17d0a261734d45866525b35de7cba3 already in progress: creating tablet
W20240506 19:56:49.223825 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c5f3def738fb4cb195d08e48d7d2b293 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5f3def738fb4cb195d08e48d7d2b293 already in progress: creating tablet
W20240506 19:56:49.224056 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8689ec0a8e134bceb010053b1f5a343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8689ec0a8e134bceb010053b1f5a343d already in progress: creating tablet
W20240506 19:56:49.224344 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ce6483470c864697be6d9590ec95edfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce6483470c864697be6d9590ec95edfc already in progress: creating tablet
W20240506 19:56:49.224575 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet e1d7a850a3e644c3a38ab224efcc8af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d7a850a3e644c3a38ab224efcc8af5 already in progress: creating tablet
W20240506 19:56:49.224781 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a8448ad8e9384edba7202d0bdce516a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8448ad8e9384edba7202d0bdce516a4 already in progress: creating tablet
W20240506 19:56:49.225006 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4f382a89f94e4fdf9b8266d049635329 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f382a89f94e4fdf9b8266d049635329 already in progress: creating tablet
I20240506 19:56:49.225414 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7397db1e74b4488a3f9a70136bccb with delete_type TABLET_DATA_DELETED (replaced by tablet 65315d8ac5ff408bae8593681335e5b1) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.225632 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.192239 13242 rpcz_store.cc:276] 0506 19:56:44.906871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.906931 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:49.191947 (+4285016us) ts_tablet_manager.cc:1120] Deleting tablet 04738def56324a368f8182543107e32e
0506 19:56:49.191961 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.192058 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4284796}
I20240506 19:56:49.228404 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 9e3d13322b4a4a2db3b6d425679f68ed (table default.Table_d9668767a7184407ac5f03e85864411e [id=e3a4e004e5c64784afdcc92d35a67b5f]) successfully deleted
I20240506 19:56:49.228478 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 580) took 4320 ms. Trace:
I20240506 19:56:49.228685 13242 rpcz_store.cc:276] 0506 19:56:44.908141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.908192 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:49.228326 (+4320134us) ts_tablet_manager.cc:1120] Deleting tablet 03df4632adfa4b0ab999aeaa5bb4e1f4
0506 19:56:49.228337 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.228456 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4319926}
I20240506 19:56:49.229117 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 582) took 4300 ms. Trace:
I20240506 19:56:49.229233 13242 rpcz_store.cc:276] 0506 19:56:44.928742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.928806 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:49.229020 (+4300214us) ts_tablet_manager.cc:1120] Deleting tablet 09cb71b5962f4561bf0fdbe12bf45296
0506 19:56:49.229030 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.229103 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4299930}
I20240506 19:56:49.229635 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 583) took 4298 ms. Trace:
W20240506 19:56:49.229650 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 04738def56324a368f8182543107e32e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04738def56324a368f8182543107e32e already in progress: creating tablet
I20240506 19:56:49.229777 13242 rpcz_store.cc:276] 0506 19:56:44.931464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.931511 (+ 47us) service_pool.cc:225] Handling call
0506 19:56:49.229506 (+4297995us) ts_tablet_manager.cc:1120] Deleting tablet ca23d6f0a9e0445cbd4d0e708c8de2d3
0506 19:56:49.229515 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.229597 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4297769}
I20240506 19:56:49.230108 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 581) took 4302 ms. Trace:
W20240506 19:56:49.230216 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 already in progress: creating tablet
I20240506 19:56:49.230190 13242 rpcz_store.cc:276] 0506 19:56:44.927194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.935061 (+ 7867us) service_pool.cc:225] Handling call
0506 19:56:49.229979 (+4294918us) ts_tablet_manager.cc:1120] Deleting tablet 50ef5a2799b447f992207452dbe39645
0506 19:56:49.229987 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.230056 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4294698}
W20240506 19:56:49.230427 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 09cb71b5962f4561bf0fdbe12bf45296 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09cb71b5962f4561bf0fdbe12bf45296 already in progress: creating tablet
I20240506 19:56:49.230523 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 584) took 4273 ms. Trace:
W20240506 19:56:49.230643 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca23d6f0a9e0445cbd4d0e708c8de2d3 already in progress: creating tablet
I20240506 19:56:49.230609 13242 rpcz_store.cc:276] 0506 19:56:44.956997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:44.968952 (+ 11955us) service_pool.cc:225] Handling call
0506 19:56:49.230395 (+4261443us) ts_tablet_manager.cc:1120] Deleting tablet 9df3ab9dff5148f9bb335a7209b29bbc
0506 19:56:49.230406 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.230475 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4261165}
I20240506 19:56:49.230896 13242 tablet_replica.cc:330] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.231168 13242 raft_consensus.cc:2229] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.231374 13242 raft_consensus.cc:2258] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:49.231626 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 50ef5a2799b447f992207452dbe39645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ef5a2799b447f992207452dbe39645 already in progress: creating tablet
W20240506 19:56:49.231868 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9df3ab9dff5148f9bb335a7209b29bbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9df3ab9dff5148f9bb335a7209b29bbc already in progress: creating tablet
I20240506 19:56:49.233119 13244 tablet_bootstrap.cc:492] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.233274 13244 ts_tablet_manager.cc:1367] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.006s
I20240506 19:56:49.234097 13244 raft_consensus.cc:348] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.234318 13244 raft_consensus.cc:374] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.234426 13244 raft_consensus.cc:729] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.234684 13244 consensus_queue.cc:260] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:49.236263 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a03f574aec984b83ab596e92a6f323fc consensus state has no leader
I20240506 19:56:49.236282 13244 ts_tablet_manager.cc:1398] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:49.243459 13243 ts_tablet_manager.cc:1875] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:49.250792 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d0d827d732e4261a7a8edb549c1c8e3 consensus state has no leader
I20240506 19:56:49.251464 13249 tablet_bootstrap.cc:492] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.251626 13249 ts_tablet_manager.cc:1367] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.223s user 0.008s sys 0.004s
I20240506 19:56:49.255088 12842 raft_consensus.cc:3037] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.258914 13249 raft_consensus.cc:348] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.259794 13249 raft_consensus.cc:374] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.259868 13249 raft_consensus.cc:729] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.260110 13249 consensus_queue.cc:260] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.261588 13249 ts_tablet_manager.cc:1398] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240506 19:56:49.262138 13249 tablet_bootstrap.cc:492] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.264279 13249 tablet_bootstrap.cc:654] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.264425 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "378759ca310f453fba45d48879e2dd1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:49.264658 13120 raft_consensus.cc:3037] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.267201 13120 raft_consensus.cc:2454] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:49.268057 14080 raft_consensus.cc:686] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:49.269541 14080 consensus_queue.cc:237] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.272436 13246 ts_tablet_manager.cc:1888] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.272580 13246 log.cc:1196] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/fdab4110236e4f5c8690f6d963ff27f7
I20240506 19:56:49.274709 13246 ts_tablet_manager.cc:1909] T fdab4110236e4f5c8690f6d963ff27f7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.274966 14143 raft_consensus.cc:482] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.275451 14143 raft_consensus.cc:504] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.275574 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 731) took 1214 ms. Trace:
I20240506 19:56:49.275671 13246 rpcz_store.cc:276] 0506 19:56:48.061284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.062218 (+ 934us) service_pool.cc:225] Handling call
0506 19:56:49.217783 (+1155565us) ts_tablet_manager.cc:1120] Deleting tablet fdab4110236e4f5c8690f6d963ff27f7
0506 19:56:49.217793 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.266059 (+ 48266us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.272319 (+ 6260us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.275551 (+ 3232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155428}
I20240506 19:56:49.275856 13245 tablet_bootstrap.cc:492] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.275969 13245 ts_tablet_manager.cc:1367] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.134s user 0.043s sys 0.030s
I20240506 19:56:49.275990 13246 tablet_replica.cc:330] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.276152 14143 leader_election.cc:288] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.276175 13246 raft_consensus.cc:2229] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.276304 13246 raft_consensus.cc:2258] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.276232 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet fdab4110236e4f5c8690f6d963ff27f7 (table default.Table_b4ce4fd613b049df9285a95fe23671c1 [id=89681fb9b58c4f429a4f94873f3827ff]) successfully deleted
I20240506 19:56:49.276629 13245 raft_consensus.cc:348] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.276793 13246 ts_tablet_manager.cc:1875] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.276806 13245 raft_consensus.cc:374] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.277011 13245 raft_consensus.cc:729] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.277139 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.277210 13245 consensus_queue.cc:260] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.277695 13245 ts_tablet_manager.cc:1398] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:49.277928 13245 tablet_bootstrap.cc:492] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:49.286144 12921 leader_election.cc:341] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.288408 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bafd3c4706f74e7da36a8f3538760f43" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.288511 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.288826 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e8b8be953a4a5784f2d098fd6364e3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.290361 13245 tablet_bootstrap.cc:654] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.290843 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:49.291157 13120 raft_consensus.cc:2454] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:49.293646 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_516b51665d1d453491f1c9caf37dda83" } modify_external_catalogs: true
I20240506 19:56:49.293924 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_516b51665d1d453491f1c9caf37dda83" } modify_external_catalogs: true
W20240506 19:56:49.295639 13057 leader_election.cc:341] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.295764 13057 leader_election.cc:302] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.295974 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cd37c88af8ee458ea4d03e8901f98d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.296151 14042 raft_consensus.cc:2726] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:49.296664 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd37c88af8ee458ea4d03e8901f98d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.298255 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef" } modify_external_catalogs: true
I20240506 19:56:49.298496 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e49f2db26f4c4d249d74e7c41bcaa8ef" } modify_external_catalogs: true
I20240506 19:56:49.299237 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_be02490c0e0c43c29452e03dd1d15be0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.299814 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be02490c0e0c43c29452e03dd1d15be0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:49.300482 13057 leader_election.cc:341] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.300611 13057 leader_election.cc:302] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.301034 14042 raft_consensus.cc:2726] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:49.301769 13057 leader_election.cc:341] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.301868 13057 leader_election.cc:302] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.301867 13243 ts_tablet_manager.cc:1888] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:49.301991 13243 log.cc:1196] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ac9ecd2e034e49e49772db7435da3c81
I20240506 19:56:49.302202 14042 raft_consensus.cc:2726] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.302425 13243 ts_tablet_manager.cc:1909] T ac9ecd2e034e49e49772db7435da3c81 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:49.302845 13057 consensus_peers.cc:485] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 796628a0859c4511a8cb35e495dc8886. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:49.303309 12781 leader_election.cc:341] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:49.303351 12922 leader_election.cc:302] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.303664 13245 tablet_bootstrap.cc:492] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.303790 13245 ts_tablet_manager.cc:1367] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.010s
I20240506 19:56:49.304514 13245 raft_consensus.cc:348] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.304765 13245 raft_consensus.cc:374] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.304831 13245 raft_consensus.cc:729] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.305029 13245 consensus_queue.cc:260] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.305600 13245 ts_tablet_manager.cc:1398] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:49.305686 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b43eacefdd54cc98fa204f64ee70d7e" } new_table_name: "default.Table_f2e21527b828489cab8c6dc854d0f813" modify_external_catalogs: true
I20240506 19:56:49.305958 13245 tablet_bootstrap.cc:492] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.307057 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9197190885234701b5933420d98f8ca8" } modify_external_catalogs: true
I20240506 19:56:49.307233 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9197190885234701b5933420d98f8ca8" } modify_external_catalogs: true
I20240506 19:56:49.307358 13245 tablet_bootstrap.cc:654] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.308120 12781 leader_election.cc:302] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.308992 13242 ts_tablet_manager.cc:1875] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.309983 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9aeb8fd99d7d4b9b8958a8698a439070" } new_table_name: "default.Table_38edce058035412e8151f9b6dae55a46" modify_external_catalogs: true
I20240506 19:56:49.310375 14069 raft_consensus.cc:2726] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.313987 13245 tablet_bootstrap.cc:492] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.314154 13245 ts_tablet_manager.cc:1367] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240506 19:56:49.314884 13245 raft_consensus.cc:348] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.316533 13245 raft_consensus.cc:374] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.316506 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]). Expected version 3 got 0
I20240506 19:56:49.316594 13245 raft_consensus.cc:729] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.316794 13245 consensus_queue.cc:260] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.317407 13245 ts_tablet_manager.cc:1398] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240506 19:56:49.319599 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f63e8e12fd3b453a96413f1fc769fb9f
I20240506 19:56:49.320986 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet f63e8e12fd3b453a96413f1fc769fb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.321962 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet f63e8e12fd3b453a96413f1fc769fb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.322140 13245 tablet_bootstrap.cc:492] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.322436 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet f63e8e12fd3b453a96413f1fc769fb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:49.323984 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad0f74c7c4cd42d293fa244de3fd693b consensus state has no leader
I20240506 19:56:49.324069 13245 tablet_bootstrap.cc:654] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.326640 13244 tablet_bootstrap.cc:492] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.328325 13244 tablet_bootstrap.cc:654] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.328382 13242 ts_tablet_manager.cc:1888] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.328511 13242 log.cc:1196] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/5988b89cb52e4f858b52862138bed025
I20240506 19:56:49.328563 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_664bb720f8a04b2baf81c73ca7d3173d" } modify_external_catalogs: true
I20240506 19:56:49.328806 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_664bb720f8a04b2baf81c73ca7d3173d" } modify_external_catalogs: true
I20240506 19:56:49.328888 13242 ts_tablet_manager.cc:1909] T 5988b89cb52e4f858b52862138bed025 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.329818 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_424aab7141bd4c80967b1104df604c32" } new_table_name: "default.Table_9765ef1ea1284910aea85d1d0de68bb3" modify_external_catalogs: true
I20240506 19:56:49.330267 14143 raft_consensus.cc:2781] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.330423 14143 raft_consensus.cc:686] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:49.330664 14143 consensus_queue.cc:237] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.331600 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_632142c0692a45b69cba3c19adb0b444" } modify_external_catalogs: true
I20240506 19:56:49.331789 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_632142c0692a45b69cba3c19adb0b444" } modify_external_catalogs: true
I20240506 19:56:49.332551 13245 tablet_bootstrap.cc:492] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.332684 13245 ts_tablet_manager.cc:1367] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240506 19:56:49.333465 13245 raft_consensus.cc:348] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.333649 13245 raft_consensus.cc:374] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.333709 13245 raft_consensus.cc:729] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.333926 13245 consensus_queue.cc:260] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.334470 13245 ts_tablet_manager.cc:1398] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:49.334746 13245 tablet_bootstrap.cc:492] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.336042 13245 tablet_bootstrap.cc:654] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.336778 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_491b0792c74c4700b9c1b789c3ad16d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.337620 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_491b0792c74c4700b9c1b789c3ad16d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.340155 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_59e7933756ae440cb5377ffc3ca93c29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.329250 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 596) took 4180 ms. Trace:
I20240506 19:56:49.340584 13242 rpcz_store.cc:276] 0506 19:56:45.149014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.149078 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:49.230808 (+4081730us) ts_tablet_manager.cc:1120] Deleting tablet 5988b89cb52e4f858b52862138bed025
0506 19:56:49.230816 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.325346 (+ 94530us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.328232 (+ 2886us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.329202 (+ 970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4081458}
W20240506 19:56:49.340744 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59e7933756ae440cb5377ffc3ca93c29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.340999 13242 tablet_replica.cc:330] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.341225 13242 raft_consensus.cc:2229] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.341209 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 5988b89cb52e4f858b52862138bed025 (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]) successfully deleted
I20240506 19:56:49.341426 13242 raft_consensus.cc:2258] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.342048 13242 ts_tablet_manager.cc:1875] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.342903 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ac9ecd2e034e49e49772db7435da3c81 (table default.Table_9608d1af549c42e28d02df6bd141adf8 [id=feed002cdb9d4296a14ca76514e18ed4]) successfully deleted
I20240506 19:56:49.346789 14197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.48.252:36867 every 8 connection(s)
W20240506 19:56:49.347121 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 08535fa1600b41ff919fefa93cfc4c78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08535fa1600b41ff919fefa93cfc4c78 already in progress: creating tablet
I20240506 19:56:49.347183 13885 rpc_server.cc:307] RPC server started. Bound to: 127.12.48.252:36867
W20240506 19:56:49.348219 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet fdef0a40f2e346a798b01e83a960de09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdef0a40f2e346a798b01e83a960de09 already in progress: creating tablet
W20240506 19:56:49.349174 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d6edff8e7e6945aa926aa1486cbbc2bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6edff8e7e6945aa926aa1486cbbc2bc already in progress: creating tablet
I20240506 19:56:49.349524 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-2/data/info.pb
W20240506 19:56:49.351006 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 63f56d2180b1472a8168fb3a63589065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63f56d2180b1472a8168fb3a63589065 already in progress: creating tablet
W20240506 19:56:49.351922 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 79592a4ecacb4324b2278e1b9a75496c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79592a4ecacb4324b2278e1b9a75496c already in progress: creating tablet
I20240506 19:56:49.352257 13243 tablet_replica.cc:330] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:49.352524 13243 raft_consensus.cc:2229] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:49.352825 13243 pending_rounds.cc:62] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:49.352916 13243 pending_rounds.cc:68] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744076621688832 op_type: NO_OP noop_request { }
I20240506 19:56:49.353086 13243 raft_consensus.cc:2866] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:49.353226 13243 raft_consensus.cc:2258] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:49.353461 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb4945e5757c401cacb39037459e2b57 consensus state has no leader
W20240506 19:56:49.353674 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 50343.6us
I20240506 19:56:49.354012 12922 leader_election.cc:302] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.354624 14143 raft_consensus.cc:2781] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:49.354743 14143 raft_consensus.cc:482] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:49.354849 14143 raft_consensus.cc:3037] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.355473 12519 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:49.356369 12483 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu as pid 13885
I20240506 19:56:49.358475 14143 raft_consensus.cc:504] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.359616 14143 leader_election.cc:288] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.359894 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:49.343796 13246 ts_tablet_manager.cc:1888] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.363898 13246 log.cc:1196] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:49.364233 13246 ts_tablet_manager.cc:1909] T 2b5a46b91cb742a994496648fbf87aa4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.365242 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 732) took 1303 ms. Trace:
W20240506 19:56:49.360401 12921 leader_election.cc:341] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.369103 14042 consensus_queue.cc:1035] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:49.369910 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7446b0633a484cfba10f44a69dcc6969 consensus state has no leader
I20240506 19:56:49.370430 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet aae313961a914ed485f866d85785a788
I20240506 19:56:49.371616 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet aae313961a914ed485f866d85785a788 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.373757 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet aae313961a914ed485f866d85785a788 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.374082 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:49.374274 13120 raft_consensus.cc:3037] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.375010 12519 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:49.365382 13246 rpcz_store.cc:276] 0506 19:56:48.061393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.062427 (+ 1034us) service_pool.cc:225] Handling call
0506 19:56:49.275941 (+1213514us) ts_tablet_manager.cc:1120] Deleting tablet 2b5a46b91cb742a994496648fbf87aa4
0506 19:56:49.275950 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.284859 (+ 8909us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.321729 (+ 36870us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.365209 (+ 43480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213398}
I20240506 19:56:49.377027 13246 tablet_replica.cc:330] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.377105 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2b5a46b91cb742a994496648fbf87aa4 (table default.Table_3a6afcad38024709a1f104fa5bbadcff [id=2249bb6d1f824bacbd5b5b1c86e58c48]) successfully deleted
I20240506 19:56:49.377722 14042 raft_consensus.cc:482] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.377875 14042 raft_consensus.cc:504] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.378656 14042 leader_election.cc:288] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.379217 13120 raft_consensus.cc:2454] T b35231d3f1bb4348aa7af3c71b809cd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:49.379897 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7397db1e74b4488a3f9a70136bccb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:49.380569 13058 leader_election.cc:341] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.381502 13246 raft_consensus.cc:2229] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:49.381815 13246 pending_rounds.cc:62] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:49.381876 13246 pending_rounds.cc:68] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744073113702400 op_type: NO_OP noop_request { }
I20240506 19:56:49.382005 13246 raft_consensus.cc:2866] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:49.382100 13246 raft_consensus.cc:2258] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.386260 13243 ts_tablet_manager.cc:1875] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:49.388036 13058 consensus_peers.cc:485] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:49.389276 13244 tablet_bootstrap.cc:492] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.389767 13244 ts_tablet_manager.cc:1367] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
I20240506 19:56:49.390789 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet aae313961a914ed485f866d85785a788 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.391335 13244 raft_consensus.cc:348] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.391647 13244 raft_consensus.cc:374] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.391772 13244 raft_consensus.cc:729] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.392156 13244 consensus_queue.cc:260] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.393884 13244 ts_tablet_manager.cc:1398] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:49.394263 13244 tablet_bootstrap.cc:492] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.394817 12922 leader_election.cc:302] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.395020 13246 ts_tablet_manager.cc:1875] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.395359 14142 raft_consensus.cc:2781] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.395475 14142 raft_consensus.cc:686] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:49.395707 14142 consensus_queue.cc:237] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.407966 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7397db1e74b4488a3f9a70136bccb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.408206 12985 raft_consensus.cc:2454] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:49.412355 13249 tablet_bootstrap.cc:492] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.412501 13249 ts_tablet_manager.cc:1367] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.150s user 0.002s sys 0.006s
I20240506 19:56:49.413359 13249 raft_consensus.cc:348] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.413614 13249 raft_consensus.cc:374] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.413686 13249 raft_consensus.cc:729] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.414109 13057 leader_election.cc:302] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.414559 14042 raft_consensus.cc:2781] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:49.413949 13249 consensus_queue.cc:260] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.414649 14042 raft_consensus.cc:482] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:49.414714 14042 raft_consensus.cc:3037] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.415063 13249 ts_tablet_manager.cc:1398] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:49.415328 13249 tablet_bootstrap.cc:492] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.416698 13249 tablet_bootstrap.cc:654] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.417937 14042 raft_consensus.cc:504] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.418005 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet f4e4be565e2d4c59b3587baed72e4a9c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:49.418812 14042 leader_election.cc:288] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.419548 13246 ts_tablet_manager.cc:1888] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:49.419687 13246 log.cc:1196] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b2a5a5e193f245de96cc82986e292f05
I20240506 19:56:49.419895 14042 raft_consensus.cc:482] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.420053 13246 ts_tablet_manager.cc:1909] T b2a5a5e193f245de96cc82986e292f05 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.420058 14042 raft_consensus.cc:504] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.421387 14042 leader_election.cc:288] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:49.421855 13057 consensus_peers.cc:485] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd57563cc6c140c184d1725a2da6a475. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:49.422509 14042 raft_consensus.cc:482] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.422675 14042 raft_consensus.cc:504] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.423605 14042 leader_election.cc:288] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.424255 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7397db1e74b4488a3f9a70136bccb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:49.424444 12984 raft_consensus.cc:3037] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.424741 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6e0126d9854215a0e174a76c0a0391" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.425526 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f4e4be565e2d4c59b3587baed72e4a9c
I20240506 19:56:49.426283 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e4be565e2d4c59b3587baed72e4a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 09a396f5c8684be7967be14b1b15195b) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.426679 12519 catalog_manager.cc:6554] ReplaceTablet: tablet f4e4be565e2d4c59b3587baed72e4a9c deleted and replaced by tablet 09a396f5c8684be7967be14b1b15195b
I20240506 19:56:49.426908 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 733) took 1365 ms. Trace:
I20240506 19:56:49.426986 13246 rpcz_store.cc:276] 0506 19:56:48.061489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.062567 (+ 1078us) service_pool.cc:225] Handling call
0506 19:56:49.376976 (+1314409us) ts_tablet_manager.cc:1120] Deleting tablet b2a5a5e193f245de96cc82986e292f05
0506 19:56:49.376987 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.398343 (+ 21356us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.419444 (+ 21101us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.426881 (+ 7437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314258}
I20240506 19:56:49.427327 13246 tablet_replica.cc:330] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.427361 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f56d2180b1472a8168fb3a63589065" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.427413 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e4be565e2d4c59b3587baed72e4a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 09a396f5c8684be7967be14b1b15195b) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.427510 13246 raft_consensus.cc:2229] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:49.427692 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b2a5a5e193f245de96cc82986e292f05 (table default.Table_384bed2d6d984f9d88c62381e38b6321 [id=f83571d593d64e58876888bfb31ccb28]) successfully deleted
I20240506 19:56:49.428368 12984 raft_consensus.cc:2454] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:49.429528 13246 pending_rounds.cc:62] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:49.429615 13246 pending_rounds.cc:68] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744072972230656 op_type: NO_OP noop_request { }
W20240506 19:56:49.429622 13057 leader_election.cc:341] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.429749 13246 raft_consensus.cc:2866] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:49.429841 13246 raft_consensus.cc:2258] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:49.430076 13057 leader_election.cc:341] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.430752 13057 leader_election.cc:302] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:49.430824 13246 ts_tablet_manager.cc:1875] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.431146 14042 raft_consensus.cc:2781] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.431455 13245 tablet_bootstrap.cc:492] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.431578 13245 ts_tablet_manager.cc:1367] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.002s
I20240506 19:56:49.432341 13245 raft_consensus.cc:348] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.432518 13245 raft_consensus.cc:374] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.432565 13245 raft_consensus.cc:729] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.432765 13245 consensus_queue.cc:260] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.435060 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_86885529e3ce46e792201516f327c805"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.435710 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86885529e3ce46e792201516f327c805 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.439193 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_86cd3e5b876348eca4cd2b79f045de74" } new_table_name: "default.Table_68b03f1b0b0f468494be25ef614a5b08" modify_external_catalogs: true
I20240506 19:56:49.442569 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_946dd68492dd4528a7925366e34068ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.443117 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_946dd68492dd4528a7925366e34068ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.444491 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_49169eeefc1442ce942d129a96fc7e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.445050 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49169eeefc1442ce942d129a96fc7e55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.445994 14210 raft_consensus.cc:482] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.446168 14210 raft_consensus.cc:504] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.447122 13245 ts_tablet_manager.cc:1398] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240506 19:56:49.453469 13245 tablet_bootstrap.cc:492] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.454286 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ee7c8d5a5ae641beb78728caf7fd6603" } new_table_name: "default.Table_0316621fc64c4470a110b2ef8a2540ab" modify_external_catalogs: true
I20240506 19:56:49.456068 14210 leader_election.cc:288] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.456211 14210 raft_consensus.cc:482] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.456318 14210 raft_consensus.cc:504] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:49.457530 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd901641d6844fbfba2a2336247a8611 has no consensus state
I20240506 19:56:49.457885 13245 tablet_bootstrap.cc:654] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.458039 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.458581 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.459354 14210 leader_election.cc:288] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.460655 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a7afaab4e68b4077a2876e0a0c12a73b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.461227 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7afaab4e68b4077a2876e0a0c12a73b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.463301 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0e783c507b7342758e013828bcca3d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.463857 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e783c507b7342758e013828bcca3d66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.470522 14042 raft_consensus.cc:686] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:49.470904 14042 consensus_queue.cc:237] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.472954 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034d6309de0740d0a5806b3847e334cb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.473182 12984 raft_consensus.cc:2454] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:49.474151 13057 leader_election.cc:302] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:49.474601 14211 raft_consensus.cc:2781] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:49.474679 14211 raft_consensus.cc:482] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:49.474735 14211 raft_consensus.cc:3037] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.475474 13246 ts_tablet_manager.cc:1888] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:49.475600 13246 log.cc:1196] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/cd57563cc6c140c184d1725a2da6a475
I20240506 19:56:49.475956 13246 ts_tablet_manager.cc:1909] T cd57563cc6c140c184d1725a2da6a475 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.477507 14211 raft_consensus.cc:504] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:49.477844 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88905d81f91d45d29c83a7b5df55f6f7 consensus state has no leader
I20240506 19:56:49.404002 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be929f0e72e4d22a9af2fe6b3072bbe with delete_type TABLET_DATA_DELETED (replaced by tablet 2f7f7171d06948388bc0fdb3514202ed) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.419590 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de7397db1e74b4488a3f9a70136bccb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:49.421173 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f56d2180b1472a8168fb3a63589065" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.423655 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6e0126d9854215a0e174a76c0a0391" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.478972 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034d6309de0740d0a5806b3847e334cb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:49.427453 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e4be565e2d4c59b3587baed72e4a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 09a396f5c8684be7967be14b1b15195b) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.481707 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176c61208b1c4518820a829d9f5c0a6f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.356796 13242 ts_tablet_manager.cc:1888] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.455251 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176c61208b1c4518820a829d9f5c0a6f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.457408 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034d6309de0740d0a5806b3847e334cb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.466125 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:49.482209 13249 tablet_bootstrap.cc:492] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.483084 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034d6309de0740d0a5806b3847e334cb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:49.483311 12985 raft_consensus.cc:3037] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.483356 13249 ts_tablet_manager.cc:1367] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20240506 19:56:49.484253 13249 raft_consensus.cc:348] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.484413 13249 raft_consensus.cc:374] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.484454 13249 raft_consensus.cc:729] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.484645 13249 consensus_queue.cc:260] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.485374 13249 ts_tablet_manager.cc:1398] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:49.485661 13249 tablet_bootstrap.cc:492] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.486030 12985 raft_consensus.cc:2454] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:49.487048 13242 log.cc:1196] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1667a0bbd3af4d678434b068c47cc518
I20240506 19:56:49.487355 13249 tablet_bootstrap.cc:654] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.487671 13242 ts_tablet_manager.cc:1909] T 1667a0bbd3af4d678434b068c47cc518 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.488046 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 597) took 4144 ms. Trace:
I20240506 19:56:49.488164 13242 rpcz_store.cc:276] 0506 19:56:45.343631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.343722 (+ 91us) service_pool.cc:225] Handling call
0506 19:56:49.340904 (+3997182us) ts_tablet_manager.cc:1120] Deleting tablet 1667a0bbd3af4d678434b068c47cc518
0506 19:56:49.340918 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.353719 (+ 12801us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.356665 (+ 2946us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.487979 (+131314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3996868}
I20240506 19:56:49.488852 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1667a0bbd3af4d678434b068c47cc518 (table default.Table_dfd40b6029db498e870d3e8a44ae6bdd [id=c4a20cff71d7496e8b07f477a0c1ea1d]) successfully deleted
I20240506 19:56:49.489621 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 599) took 4097 ms. Trace:
I20240506 19:56:49.489727 13242 rpcz_store.cc:276] 0506 19:56:45.391847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.391900 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:49.489436 (+4097536us) ts_tablet_manager.cc:1120] Deleting tablet f68f57002f8144caaaf37c2838f98e1f
0506 19:56:49.489447 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.489544 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4097305}
I20240506 19:56:49.489894 13244 tablet_bootstrap.cc:654] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:49.490209 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f68f57002f8144caaaf37c2838f98e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f68f57002f8144caaaf37c2838f98e1f already in progress: creating tablet
I20240506 19:56:49.490604 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 598) took 4098 ms. Trace:
I20240506 19:56:49.490689 13242 rpcz_store.cc:276] 0506 19:56:45.391712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.397091 (+ 5379us) service_pool.cc:225] Handling call
0506 19:56:49.490462 (+4093371us) ts_tablet_manager.cc:1120] Deleting tablet da06187c83b94d03ae4ebb67a2873cf0
0506 19:56:49.490472 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.490552 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4093079}
W20240506 19:56:49.491127 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet da06187c83b94d03ae4ebb67a2873cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da06187c83b94d03ae4ebb67a2873cf0 already in progress: creating tablet
I20240506 19:56:49.493140 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 600) took 4087 ms. Trace:
I20240506 19:56:49.493238 13242 rpcz_store.cc:276] 0506 19:56:45.406104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.406183 (+ 79us) service_pool.cc:225] Handling call
0506 19:56:49.492960 (+4086777us) ts_tablet_manager.cc:1120] Deleting tablet 99f4861a01404663870ce91978c8af4f
0506 19:56:49.492971 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.493073 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4084909}
W20240506 19:56:49.493726 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 99f4861a01404663870ce91978c8af4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99f4861a01404663870ce91978c8af4f already in progress: creating tablet
I20240506 19:56:49.494354 13242 tablet_replica.cc:330] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.494575 13242 raft_consensus.cc:2229] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.494767 13242 raft_consensus.cc:2258] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.495322 13242 ts_tablet_manager.cc:1875] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.500419 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet b3e50bde3564493eb7267acb42cb8592 (table default.Table_95be5bd2c51b4613b0152034aae6350d [id=b8f3e6ecf02f4dd299adfa1d7464e53f]). Expected version 1 got 0
I20240506 19:56:49.500844 12521 catalog_manager.cc:5484] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:49.502933 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 910d1465df784dfba6d78ff7fd9f2722 (DEFAULT_TABLE table=default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.503355 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d1465df784dfba6d78ff7fd9f2722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.503551 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 910d1465df784dfba6d78ff7fd9f2722 (DEFAULT_TABLE table=default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.503947 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d1465df784dfba6d78ff7fd9f2722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.504801 14069 raft_consensus.cc:482] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.504964 14069 raft_consensus.cc:504] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.505393 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:49.505648 14069 leader_election.cc:288] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:49.509220 12783 leader_election.cc:341] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f0a8b12f36e74c95ad6eaf30e432f3e9
I20240506 19:56:49.509922 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:49.506242 14199 sys_catalog.cc:263] Verifying existing consensus state
W20240506 19:56:49.517541 12781 leader_election.cc:341] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f0a8b12f36e74c95ad6eaf30e432f3e9
I20240506 19:56:49.517714 12781 leader_election.cc:302] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.518029 14069 raft_consensus.cc:2726] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.518718 14069 raft_consensus.cc:482] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.518891 14069 raft_consensus.cc:504] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.519548 14069 leader_election.cc:288] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.519869 14069 raft_consensus.cc:482] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.520010 14069 raft_consensus.cc:504] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.520596 14069 leader_election.cc:288] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.522469 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f7049cb5beca4e30b1f0a315789a1e1e" } modify_external_catalogs: true
I20240506 19:56:49.522783 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f7049cb5beca4e30b1f0a315789a1e1e" } modify_external_catalogs: true
I20240506 19:56:49.522879 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_803e3127dd644ef7a78b9e24616a7d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.523420 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 3fb398132c58489fb0a28492302e4a23
W20240506 19:56:49.523535 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_803e3127dd644ef7a78b9e24616a7d66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:49.525068 12783 leader_election.cc:341] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 0114edac976844619e438c17bb959010
I20240506 19:56:49.525372 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb398132c58489fb0a28492302e4a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.525435 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb398132c58489fb0a28492302e4a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.526286 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b1f83bed002d4d3fbf1a4b5b95f0ea54" } new_table_name: "default.Table_e9acd0a6178643ba98580f34636cd25b" modify_external_catalogs: true
I20240506 19:56:49.529280 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:49.530405 12781 leader_election.cc:341] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 0114edac976844619e438c17bb959010
I20240506 19:56:49.530547 12781 leader_election.cc:302] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.530702 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 58c88f80f5b34672b7c3a4b604469bf1
I20240506 19:56:49.531203 14069 raft_consensus.cc:2726] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.531731 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c88f80f5b34672b7c3a4b604469bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.532841 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_52654f2617b946ad99d761a8a1b45c4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.533471 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52654f2617b946ad99d761a8a1b45c4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.533550 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c88f80f5b34672b7c3a4b604469bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.534850 13249 tablet_bootstrap.cc:492] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.535064 13249 ts_tablet_manager.cc:1367] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20240506 19:56:49.535831 12950 tablet_service.cc:1467] Processing CreateTablet for tablet ba4d19e4a76242b8a452792cd361671f (DEFAULT_TABLE table=default.Table_03fba11fcfa745868fa55d4fe134c993 [id=b4cf5ca7aeb145caaea7b343bba253b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.535914 13249 raft_consensus.cc:348] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.536109 13249 raft_consensus.cc:374] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.536175 13249 raft_consensus.cc:729] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.536248 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4d19e4a76242b8a452792cd361671f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.536398 13249 consensus_queue.cc:260] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.536485 12819 tablet_service.cc:1467] Processing CreateTablet for tablet ba4d19e4a76242b8a452792cd361671f (DEFAULT_TABLE table=default.Table_03fba11fcfa745868fa55d4fe134c993 [id=b4cf5ca7aeb145caaea7b343bba253b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.536989 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4d19e4a76242b8a452792cd361671f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.539731 13242 ts_tablet_manager.cc:1888] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.540256 13242 log.cc:1196] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d6c656e16c1045dea79bc23b64fdd5e8
I20240506 19:56:49.540820 14069 raft_consensus.cc:482] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.540915 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 9c93117bdcf44c719c2e3588de538e96 (DEFAULT_TABLE table=default.Table_5893f3f184b247949785e248aa0e53c9 [id=7ce9557d82c14955b66d3970197b5bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.540971 14069 raft_consensus.cc:504] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.541307 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c93117bdcf44c719c2e3588de538e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.541808 14069 leader_election.cc:288] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.541018 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Bootstrap starting.
I20240506 19:56:49.542250 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 9c93117bdcf44c719c2e3588de538e96 (DEFAULT_TABLE table=default.Table_5893f3f184b247949785e248aa0e53c9 [id=7ce9557d82c14955b66d3970197b5bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.542440 13242 ts_tablet_manager.cc:1909] T d6c656e16c1045dea79bc23b64fdd5e8 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.542645 13244 tablet_bootstrap.cc:492] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.542771 13244 ts_tablet_manager.cc:1367] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.000s
W20240506 19:56:49.543540 12781 leader_election.cc:341] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f3e7329b22414da886dee4e1ec6bf3ad
I20240506 19:56:49.543617 13244 raft_consensus.cc:348] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.543777 13244 raft_consensus.cc:374] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.543817 13244 raft_consensus.cc:729] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.543974 13244 consensus_queue.cc:260] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.544409 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 601) took 4136 ms. Trace:
I20240506 19:56:49.545174 13243 ts_tablet_manager.cc:1888] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:49.545265 13243 log.cc:1196] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/378759ca310f453fba45d48879e2dd1a
I20240506 19:56:49.544677 13242 rpcz_store.cc:276] 0506 19:56:45.407500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.407929 (+ 429us) service_pool.cc:225] Handling call
0506 19:56:49.493994 (+4086065us) ts_tablet_manager.cc:1120] Deleting tablet d6c656e16c1045dea79bc23b64fdd5e8
0506 19:56:49.494003 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.522351 (+ 28348us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.539200 (+ 16849us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.544275 (+ 5075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4085878}
W20240506 19:56:49.479583 13058 leader_election.cc:341] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.545964 13058 leader_election.cc:302] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:49.546257 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 66682.3us
W20240506 19:56:49.546450 13058 leader_election.cc:341] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:49.546476 12783 leader_election.cc:341] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f3e7329b22414da886dee4e1ec6bf3ad
I20240506 19:56:49.546528 13058 leader_election.cc:302] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.546572 12783 leader_election.cc:302] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:49.546856 13058 leader_election.cc:341] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.546980 14069 raft_consensus.cc:2726] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.547227 14210 raft_consensus.cc:2726] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.547390 14210 raft_consensus.cc:2726] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.491067 13057 leader_election.cc:302] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:49.547727 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 56698.6us
I20240506 19:56:49.547737 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d6c656e16c1045dea79bc23b64fdd5e8 (table default.Table_c905679471c7485fa59180f110b5ec1f [id=c4769761819944109eebf88cb916cb27]) successfully deleted
I20240506 19:56:49.547817 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 602) took 4137 ms. Trace:
W20240506 19:56:49.547992 13057 leader_election.cc:341] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.513880 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 with delete_type TABLET_DATA_DELETED (replaced by tablet 986f9c83b04f41b49e9b78fce731a90f) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.514091 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet f8951bf4873e4382bcd42a52368fb376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.514333 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.548470 14210 raft_consensus.cc:2781] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.548094 13242 rpcz_store.cc:276] 0506 19:56:45.410052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.410621 (+ 569us) service_pool.cc:225] Handling call
0506 19:56:49.545599 (+4134978us) ts_tablet_manager.cc:1120] Deleting tablet 55d9f32618b64ffa92b5a868081c36a6
0506 19:56:49.545609 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.547711 (+ 2102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4134684}
I20240506 19:56:49.520545 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae313961a914ed485f866d85785a788" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:49.526286 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb398132c58489fb0a28492302e4a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.533994 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c88f80f5b34672b7c3a4b604469bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.539113 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 910d1465df784dfba6d78ff7fd9f2722 (DEFAULT_TABLE table=default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:49.549333 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 55d9f32618b64ffa92b5a868081c36a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55d9f32618b64ffa92b5a868081c36a6 already in progress: creating tablet
I20240506 19:56:49.549582 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d1465df784dfba6d78ff7fd9f2722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.549670 12783 leader_election.cc:302] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.550021 14069 raft_consensus.cc:2726] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.550740 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 603) took 4119 ms. Trace:
I20240506 19:56:49.550992 13242 rpcz_store.cc:276] 0506 19:56:45.430895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.430977 (+ 82us) service_pool.cc:225] Handling call
0506 19:56:49.550352 (+4119375us) ts_tablet_manager.cc:1120] Deleting tablet cd57563cc6c140c184d1725a2da6a475
0506 19:56:49.550449 (+ 97us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.550722 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4119083}
W20240506 19:56:49.551824 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet cd57563cc6c140c184d1725a2da6a475 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd57563cc6c140c184d1725a2da6a475 already in progress: creating tablet
I20240506 19:56:49.552629 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 604) took 4043 ms. Trace:
I20240506 19:56:49.552881 13242 rpcz_store.cc:276] 0506 19:56:45.508813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.508880 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:49.552253 (+4043373us) ts_tablet_manager.cc:1120] Deleting tablet ea7e073296ce46e786362aaed96eb4dd
0506 19:56:49.552349 (+ 96us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.552613 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4043097}
I20240506 19:56:49.516511 13245 tablet_bootstrap.cc:492] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:49.553351 13058 leader_election.cc:341] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.553469 13245 ts_tablet_manager.cc:1367] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
W20240506 19:56:49.553678 13058 leader_election.cc:341] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:49.553761 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ea7e073296ce46e786362aaed96eb4dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7e073296ce46e786362aaed96eb4dd already in progress: creating tablet
I20240506 19:56:49.553768 13058 leader_election.cc:302] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.554275 13245 raft_consensus.cc:348] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.554322 13242 tablet_replica.cc:330] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.554457 13245 raft_consensus.cc:374] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.554509 13245 raft_consensus.cc:729] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.554703 13245 consensus_queue.cc:260] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.539723 13103 tablet_service.cc:1467] Processing CreateTablet for tablet ba4d19e4a76242b8a452792cd361671f (DEFAULT_TABLE table=default.Table_03fba11fcfa745868fa55d4fe134c993 [id=b4cf5ca7aeb145caaea7b343bba253b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.554981 13242 raft_consensus.cc:2229] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.555394 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4d19e4a76242b8a452792cd361671f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.485440 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 734) took 1423 ms. Trace:
I20240506 19:56:49.555507 13242 raft_consensus.cc:2258] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.555591 13246 rpcz_store.cc:276] 0506 19:56:48.061581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.062745 (+ 1164us) service_pool.cc:225] Handling call
0506 19:56:49.427272 (+1364527us) ts_tablet_manager.cc:1120] Deleting tablet cd57563cc6c140c184d1725a2da6a475
0506 19:56:49.427283 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.434053 (+ 6770us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.475363 (+ 41310us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.485415 (+ 10052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364401}
I20240506 19:56:49.555886 13246 tablet_replica.cc:330] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.556042 13246 raft_consensus.cc:2229] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.556165 13246 raft_consensus.cc:2258] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.556664 13246 ts_tablet_manager.cc:1875] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.557499 13242 ts_tablet_manager.cc:1875] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.559634 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.565593 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet cd57563cc6c140c184d1725a2da6a475 (table default.Table_ff179844ac74496183bb76ddcabbcc75 [id=6a6084dba9e54d238890a1c9be17fdec]) successfully deleted
I20240506 19:56:49.575490 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 2271dc77d307453683ceb193bd2fd71d (DEFAULT_TABLE table=default.Table_aff3a2a3bcf74711a29fc36b829be54d [id=77bf1f6b60274489b239a5270fb3d76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.576066 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2271dc77d307453683ceb193bd2fd71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.576660 14042 raft_consensus.cc:2726] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.578037 13242 ts_tablet_manager.cc:1888] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.578437 13242 log.cc:1196] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/4f91888e88234cbdbd00f48dc1574653
I20240506 19:56:49.579850 13242 ts_tablet_manager.cc:1909] T 4f91888e88234cbdbd00f48dc1574653 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.580101 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 9c93117bdcf44c719c2e3588de538e96 (DEFAULT_TABLE table=default.Table_5893f3f184b247949785e248aa0e53c9 [id=7ce9557d82c14955b66d3970197b5bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.580543 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c93117bdcf44c719c2e3588de538e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.581430 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 609) took 3976 ms. Trace:
I20240506 19:56:49.581827 13242 rpcz_store.cc:276] 0506 19:56:45.604639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.605350 (+ 711us) service_pool.cc:225] Handling call
0506 19:56:49.554183 (+3948833us) ts_tablet_manager.cc:1120] Deleting tablet 4f91888e88234cbdbd00f48dc1574653
0506 19:56:49.554280 (+ 97us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.568110 (+ 13830us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.577659 (+ 9549us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.581198 (+ 3539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3948620}
I20240506 19:56:49.582839 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 4f91888e88234cbdbd00f48dc1574653 (table default.Table_4b6f407e4a464024aed5a1fb84c49849 [id=c7b8c2f7b9f74051b8207e0e127d930d]) successfully deleted
I20240506 19:56:49.584234 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 611) took 3928 ms. Trace:
I20240506 19:56:49.584642 13242 rpcz_store.cc:276] 0506 19:56:45.655379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.655638 (+ 259us) service_pool.cc:225] Handling call
0506 19:56:49.583691 (+3928053us) ts_tablet_manager.cc:1120] Deleting tablet 9c19567619b64ee0a8dd637d7f69ce3e
0506 19:56:49.583706 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.584129 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3927674}
I20240506 19:56:49.586441 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 612) took 3930 ms. Trace:
I20240506 19:56:49.586700 13242 rpcz_store.cc:276] 0506 19:56:45.655588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.655980 (+ 392us) service_pool.cc:225] Handling call
0506 19:56:49.585963 (+3929983us) ts_tablet_manager.cc:1120] Deleting tablet dc206b31d3da458cb944d5f00e23d702
0506 19:56:49.586058 (+ 95us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.586340 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3929793}
W20240506 19:56:49.587534 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet dc206b31d3da458cb944d5f00e23d702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc206b31d3da458cb944d5f00e23d702 already in progress: creating tablet
I20240506 19:56:49.588351 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 613) took 3929 ms. Trace:
I20240506 19:56:49.588604 13242 rpcz_store.cc:276] 0506 19:56:45.658628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.658679 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:49.587964 (+3929285us) ts_tablet_manager.cc:1120] Deleting tablet 9944e5bfe0a741d69b8f9b6a9dad84dd
0506 19:56:49.588058 (+ 94us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.588333 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3929058}
I20240506 19:56:49.588951 13246 ts_tablet_manager.cc:1888] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.589179 13246 log.cc:1196] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d8f3e7df09f347bfa5cad4de34dbda13
W20240506 19:56:49.589450 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9944e5bfe0a741d69b8f9b6a9dad84dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9944e5bfe0a741d69b8f9b6a9dad84dd already in progress: creating tablet
I20240506 19:56:49.589946 13246 ts_tablet_manager.cc:1909] T d8f3e7df09f347bfa5cad4de34dbda13 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.590212 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 614) took 3923 ms. Trace:
I20240506 19:56:49.590615 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 20d071e027174b03b01cf788c0b95e89 (DEFAULT_TABLE table=default.Table_73d6d75fc8f6456cba8ffeba3ecc9404 [id=a867a0cd1eca411a85db8f03ee181d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.590479 13242 rpcz_store.cc:276] 0506 19:56:45.666442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.666496 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:49.589812 (+3923316us) ts_tablet_manager.cc:1120] Deleting tablet ac9ecd2e034e49e49772db7435da3c81
0506 19:56:49.589916 (+ 104us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.590195 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3923081}
I20240506 19:56:49.591117 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d071e027174b03b01cf788c0b95e89. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.591416 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ac9ecd2e034e49e49772db7435da3c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9ecd2e034e49e49772db7435da3c81 already in progress: creating tablet
I20240506 19:56:49.592417 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 620) took 3652 ms. Trace:
I20240506 19:56:49.576341 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 2271dc77d307453683ceb193bd2fd71d (DEFAULT_TABLE table=default.Table_aff3a2a3bcf74711a29fc36b829be54d [id=77bf1f6b60274489b239a5270fb3d76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.592810 13242 rpcz_store.cc:276] 0506 19:56:45.939728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.939838 (+ 110us) service_pool.cc:225] Handling call
0506 19:56:49.591825 (+3651987us) ts_tablet_manager.cc:1120] Deleting tablet 796beb8e0a2c41d698bd4c05796263f1
0506 19:56:49.591926 (+ 101us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.592298 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3651706}
I20240506 19:56:49.590425 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 20d071e027174b03b01cf788c0b95e89 (DEFAULT_TABLE table=default.Table_73d6d75fc8f6456cba8ffeba3ecc9404 [id=a867a0cd1eca411a85db8f03ee181d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.593652 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2271dc77d307453683ceb193bd2fd71d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.593696 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 796beb8e0a2c41d698bd4c05796263f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 796beb8e0a2c41d698bd4c05796263f1 already in progress: creating tablet
I20240506 19:56:49.593847 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d071e027174b03b01cf788c0b95e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.594761 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 621) took 3639 ms. Trace:
I20240506 19:56:49.595017 13242 rpcz_store.cc:276] 0506 19:56:45.955061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:45.955122 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:49.594385 (+3639263us) ts_tablet_manager.cc:1120] Deleting tablet 159ea8e0a4ee47fea93eb4b36cdd2495
0506 19:56:49.594478 (+ 93us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.594655 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3639037}
W20240506 19:56:49.595942 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 159ea8e0a4ee47fea93eb4b36cdd2495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 159ea8e0a4ee47fea93eb4b36cdd2495 already in progress: creating tablet
I20240506 19:56:49.596931 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 622) took 3548 ms. Trace:
I20240506 19:56:49.597586 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 979d789bc89a435b9e6cd8d6dc9c2037
I20240506 19:56:49.597203 13242 rpcz_store.cc:276] 0506 19:56:46.048511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.048578 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:49.596370 (+3547792us) ts_tablet_manager.cc:1120] Deleting tablet c0c7cae21c714148a37cab389d116e91
0506 19:56:49.596473 (+ 103us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.596914 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3547479}
I20240506 19:56:49.597780 12809 tablet_service.cc:1467] Processing CreateTablet for tablet c8eeaa71ac61424796c6ffdd993de9db (DEFAULT_TABLE table=default.Table_964a0d50cb2d48fa988072257bec9ec2 [id=43ef045c7d6c44ada81c91c8d93861c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.598134 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8eeaa71ac61424796c6ffdd993de9db. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.598232 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c0c7cae21c714148a37cab389d116e91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0c7cae21c714148a37cab389d116e91 already in progress: creating tablet
I20240506 19:56:49.598690 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 979d789bc89a435b9e6cd8d6dc9c2037 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.599257 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 623) took 3534 ms. Trace:
I20240506 19:56:49.599534 13242 rpcz_store.cc:276] 0506 19:56:46.064903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.064964 (+ 61us) service_pool.cc:225] Handling call
0506 19:56:49.598790 (+3533826us) ts_tablet_manager.cc:1120] Deleting tablet 4be4e628b9364cccb525b8e1b8070286
0506 19:56:49.598893 (+ 103us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.599239 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3533600}
W20240506 19:56:49.600898 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4be4e628b9364cccb525b8e1b8070286 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be4e628b9364cccb525b8e1b8070286 already in progress: creating tablet
I20240506 19:56:49.600916 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 979d789bc89a435b9e6cd8d6dc9c2037 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.601393 12950 tablet_service.cc:1467] Processing CreateTablet for tablet c8eeaa71ac61424796c6ffdd993de9db (DEFAULT_TABLE table=default.Table_964a0d50cb2d48fa988072257bec9ec2 [id=43ef045c7d6c44ada81c91c8d93861c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.601852 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8eeaa71ac61424796c6ffdd993de9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.601953 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 624) took 3526 ms. Trace:
I20240506 19:56:49.602232 13242 rpcz_store.cc:276] 0506 19:56:46.075591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.075664 (+ 73us) service_pool.cc:225] Handling call
0506 19:56:49.601563 (+3525899us) ts_tablet_manager.cc:1120] Deleting tablet 7bc30f4690b24c5ea8ab2f7a26122992
0506 19:56:49.601573 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.601930 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3525599}
W20240506 19:56:49.603212 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7bc30f4690b24c5ea8ab2f7a26122992 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bc30f4690b24c5ea8ab2f7a26122992 already in progress: creating tablet
I20240506 19:56:49.603884 13242 tablet_replica.cc:330] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.602090 13243 ts_tablet_manager.cc:1909] T 378759ca310f453fba45d48879e2dd1a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:49.604559 13242 raft_consensus.cc:2229] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.605154 13242 raft_consensus.cc:2258] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.478624 14211 leader_election.cc:288] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:49.592981 13058 leader_election.cc:341] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.597638 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 20d071e027174b03b01cf788c0b95e89 (DEFAULT_TABLE table=default.Table_73d6d75fc8f6456cba8ffeba3ecc9404 [id=a867a0cd1eca411a85db8f03ee181d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.606122 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d071e027174b03b01cf788c0b95e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.600641 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 2271dc77d307453683ceb193bd2fd71d (DEFAULT_TABLE table=default.Table_aff3a2a3bcf74711a29fc36b829be54d [id=77bf1f6b60274489b239a5270fb3d76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.606815 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2271dc77d307453683ceb193bd2fd71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.607218 13242 ts_tablet_manager.cc:1875] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.609627 13101 tablet_service.cc:1467] Processing CreateTablet for tablet c8eeaa71ac61424796c6ffdd993de9db (DEFAULT_TABLE table=default.Table_964a0d50cb2d48fa988072257bec9ec2 [id=43ef045c7d6c44ada81c91c8d93861c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.610016 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8eeaa71ac61424796c6ffdd993de9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.610064 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c93117bdcf44c719c2e3588de538e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.619936 14211 raft_consensus.cc:482] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.620293 14211 raft_consensus.cc:504] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.622648 12961 tablet_service.cc:1467] Processing CreateTablet for tablet d6d9907f476f45d791920f2a60f976a7 (DEFAULT_TABLE table=default.Table_9c5d2e15a9e64b618f68377dce21b541 [id=ad10dbcda08d48c79953b88a4de5eaa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.622795 12809 tablet_service.cc:1467] Processing CreateTablet for tablet d6d9907f476f45d791920f2a60f976a7 (DEFAULT_TABLE table=default.Table_9c5d2e15a9e64b618f68377dce21b541 [id=ad10dbcda08d48c79953b88a4de5eaa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.623180 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d9907f476f45d791920f2a60f976a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.623179 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d9907f476f45d791920f2a60f976a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.621232 14211 leader_election.cc:288] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.630199 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:49.630358 13242 ts_tablet_manager.cc:1888] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.630383 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 744864043f9347e0a13f8c2a7d6b147f
I20240506 19:56:49.630834 13242 log.cc:1196] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b529bb9362924caca399491e7d9e03cd
I20240506 19:56:49.631814 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 744864043f9347e0a13f8c2a7d6b147f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.631867 14143 raft_consensus.cc:482] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.631953 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607b3fc2c17b4793af9b84c19066ec5e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.632038 14143 raft_consensus.cc:504] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.632135 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607b3fc2c17b4793af9b84c19066ec5e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.632251 13249 ts_tablet_manager.cc:1398] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.097s user 0.001s sys 0.001s
I20240506 19:56:49.632295 12984 raft_consensus.cc:2454] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:49.632524 13249 tablet_bootstrap.cc:492] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.632586 14143 leader_election.cc:288] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.634739 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607b3fc2c17b4793af9b84c19066ec5e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:49.635349 12921 consensus_peers.cc:485] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:49.635514 12921 leader_election.cc:341] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.636178 13057 leader_election.cc:302] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:49.544409 13244 ts_tablet_manager.cc:1398] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:49.637367 13244 tablet_bootstrap.cc:492] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.605350 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 735) took 1543 ms. Trace:
I20240506 19:56:49.638231 13246 rpcz_store.cc:276] 0506 19:56:48.061687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.062894 (+ 1207us) service_pool.cc:225] Handling call
0506 19:56:49.555843 (+1492949us) ts_tablet_manager.cc:1120] Deleting tablet d8f3e7df09f347bfa5cad4de34dbda13
0506 19:56:49.555853 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.559505 (+ 3652us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.588850 (+ 29345us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.605325 (+ 16475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492810}
I20240506 19:56:49.638621 13246 tablet_replica.cc:330] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.638828 13246 raft_consensus.cc:2229] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.638983 13246 raft_consensus.cc:2258] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.638940 13242 ts_tablet_manager.cc:1909] T b529bb9362924caca399491e7d9e03cd P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.639549 13246 ts_tablet_manager.cc:1875] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.640461 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 625) took 3506 ms. Trace:
I20240506 19:56:49.640687 13242 rpcz_store.cc:276] 0506 19:56:46.133621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.133684 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:49.603653 (+3469969us) ts_tablet_manager.cc:1120] Deleting tablet b529bb9362924caca399491e7d9e03cd
0506 19:56:49.603751 (+ 98us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.618386 (+ 14635us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.629951 (+ 11565us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.640352 (+ 10401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3468413}
I20240506 19:56:49.641808 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 979d789bc89a435b9e6cd8d6dc9c2037 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.642274 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b529bb9362924caca399491e7d9e03cd (table default.Table_5f9eb3d0b4d74469a1d3e7bfa3ce8d8c [id=fd3051399ebf4aa2b31bbd709b120d48]) successfully deleted
I20240506 19:56:49.642668 13242 tablet_replica.cc:330] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.642956 13242 raft_consensus.cc:2229] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.643203 13242 raft_consensus.cc:2258] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.644205 13249 tablet_bootstrap.cc:654] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.644536 14211 raft_consensus.cc:2781] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:49.645344 13242 ts_tablet_manager.cc:1875] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.645207 14211 raft_consensus.cc:482] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:49.646006 14211 raft_consensus.cc:3037] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.647116 14222 raft_consensus.cc:482] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.647251 14222 raft_consensus.cc:504] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.647862 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d8f3e7df09f347bfa5cad4de34dbda13 (table default.Table_d5bb6a8c548e46ecb59035f49816ad52 [id=70785b5175b34eb68dc3fcba2255645b]) successfully deleted
I20240506 19:56:49.648536 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 744864043f9347e0a13f8c2a7d6b147f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.648731 14222 leader_election.cc:288] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:49.648736 13058 leader_election.cc:341] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.649525 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8989f02f25418ea9f824b3e5eeddd6 (DEFAULT_TABLE table=default.Table_74493d529d484630900cd65b7b1777da [id=69f26966a0b84d4e9ada2962c959c584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.649955 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8989f02f25418ea9f824b3e5eeddd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.653576 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac353c6b9e541f181e20ad72775899a (DEFAULT_TABLE table=default.Table_1258c1a3736349b0973af069d7d58b02 [id=1ab3cd1ac4c744dbb1b0cb55267eea97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.653934 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac353c6b9e541f181e20ad72775899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.653991 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 71e4cd127e494acd8b7dffd87cb6bfa0 (DEFAULT_TABLE table=default.Table_36502249055f419692c8fe55e7f7d2ea [id=59360d532f304a3bbab1c482d0ba8c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.654290 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e4cd127e494acd8b7dffd87cb6bfa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.655328 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 744864043f9347e0a13f8c2a7d6b147f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.657655 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 71e4cd127e494acd8b7dffd87cb6bfa0 (DEFAULT_TABLE table=default.Table_36502249055f419692c8fe55e7f7d2ea [id=59360d532f304a3bbab1c482d0ba8c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.658053 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e4cd127e494acd8b7dffd87cb6bfa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.659348 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed6da25f3b94b26959e66e513088c16 (DEFAULT_TABLE table=default.Table_ceed94ec290a471ca0114a9727776f20 [id=54ebe9481ac5420b8adc3c32a9af518a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.659662 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed6da25f3b94b26959e66e513088c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.660782 12520 catalog_manager.cc:5484] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:49.662509 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed6da25f3b94b26959e66e513088c16 (DEFAULT_TABLE table=default.Table_ceed94ec290a471ca0114a9727776f20 [id=54ebe9481ac5420b8adc3c32a9af518a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.662875 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed6da25f3b94b26959e66e513088c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.662797 14211 raft_consensus.cc:504] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:49.664364 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 491909887be54436bb94162ace8dd78e consensus state has no leader
I20240506 19:56:49.665050 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 (DEFAULT_TABLE table=default.Table_29547e0639d84d1d803dd3f2a9fed320 [id=8c60737562ee4762a3b78a45829449e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.665457 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e64cb2f31b64b07848c2dfed18ab9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.666785 14069 raft_consensus.cc:482] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.666940 14069 raft_consensus.cc:504] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.667587 14069 leader_election.cc:288] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.667534 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:49.669384 13245 ts_tablet_manager.cc:1398] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.116s user 0.002s sys 0.006s
I20240506 19:56:49.669689 13245 tablet_bootstrap.cc:492] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:49.670209 12783 leader_election.cc:341] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 4be929f0e72e4d22a9af2fe6b3072bbe
W20240506 19:56:49.670732 12781 leader_election.cc:341] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 4be929f0e72e4d22a9af2fe6b3072bbe
I20240506 19:56:49.670825 12781 leader_election.cc:302] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.671249 13245 tablet_bootstrap.cc:654] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.671343 14069 raft_consensus.cc:2726] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.672875 14211 leader_election.cc:288] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.676766 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac353c6b9e541f181e20ad72775899a (DEFAULT_TABLE table=default.Table_1258c1a3736349b0973af069d7d58b02 [id=1ab3cd1ac4c744dbb1b0cb55267eea97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.677204 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac353c6b9e541f181e20ad72775899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.680809 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8989f02f25418ea9f824b3e5eeddd6 (DEFAULT_TABLE table=default.Table_74493d529d484630900cd65b7b1777da [id=69f26966a0b84d4e9ada2962c959c584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.681231 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8989f02f25418ea9f824b3e5eeddd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.682024 13242 ts_tablet_manager.cc:1888] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.682439 13242 log.cc:1196] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/f3d57249f30a44dab3384092ec20b515
I20240506 19:56:49.683606 13242 ts_tablet_manager.cc:1909] T f3d57249f30a44dab3384092ec20b515 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:49.684489 12816 tablet_service.cc:1467] Processing CreateTablet for tablet ad35eed062e843d7a17b3eed22fd90d9 (DEFAULT_TABLE table=default.Table_1458eb9e9352452f88190029f7dfc864 [id=667533b391c748afb927260245b13130]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.684859 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad35eed062e843d7a17b3eed22fd90d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.686290 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 626) took 3349 ms. Trace:
I20240506 19:56:49.686650 13242 rpcz_store.cc:276] 0506 19:56:46.336665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.336736 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:49.641817 (+3305081us) ts_tablet_manager.cc:1120] Deleting tablet f3d57249f30a44dab3384092ec20b515
0506 19:56:49.642597 (+ 780us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.664918 (+ 22321us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.681760 (+ 16842us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.686268 (+ 4508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3304723}
I20240506 19:56:49.687598 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet f3d57249f30a44dab3384092ec20b515 (table default.Table_11b5bac0d01243f4a527f32ba62d5a38 [id=7197d7d0733346f4bc417c7dca54b79f]) successfully deleted
I20240506 19:56:49.688558 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 630) took 3322 ms. Trace:
I20240506 19:56:49.688827 13242 rpcz_store.cc:276] 0506 19:56:46.366414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.366652 (+ 238us) service_pool.cc:225] Handling call
0506 19:56:49.688133 (+3321481us) ts_tablet_manager.cc:1120] Deleting tablet a20016b56be94248905e0e8d6a35100f
0506 19:56:49.688144 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.688538 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3321157}
W20240506 19:56:49.690793 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a20016b56be94248905e0e8d6a35100f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20016b56be94248905e0e8d6a35100f already in progress: creating tablet
I20240506 19:56:49.691428 12817 tablet_service.cc:1467] Processing CreateTablet for tablet fee3e6be377b42818aaa7dda59a1ffda (DEFAULT_TABLE table=default.Table_aa42760da3bc4705bb6912cc7ebbfb74 [id=246f3a1786da4e438d563fa8ead150c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.692003 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 28b1e772db304a038a46d428b8c522c4 (DEFAULT_TABLE table=default.Table_112c674a2f6a46cb8e60669eb015488f [id=d50032a383514ade85a7e639245332b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.692459 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b1e772db304a038a46d428b8c522c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.693632 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 28b1e772db304a038a46d428b8c522c4 (DEFAULT_TABLE table=default.Table_112c674a2f6a46cb8e60669eb015488f [id=d50032a383514ade85a7e639245332b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.694119 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b1e772db304a038a46d428b8c522c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.694959 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3e6be377b42818aaa7dda59a1ffda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.692250 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 631) took 3325 ms. Trace:
I20240506 19:56:49.695304 13242 rpcz_store.cc:276] 0506 19:56:46.366523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.366917 (+ 394us) service_pool.cc:225] Handling call
0506 19:56:49.691741 (+3324824us) ts_tablet_manager.cc:1120] Deleting tablet 028055b9df1f4667bd2e54909690e327
0506 19:56:49.691751 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.692229 (+ 478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3324601}
W20240506 19:56:49.696544 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 028055b9df1f4667bd2e54909690e327 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 028055b9df1f4667bd2e54909690e327 already in progress: creating tablet
I20240506 19:56:49.697459 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 599f6a7b76d14c9589e5cabb04bb5f4c (DEFAULT_TABLE table=default.Table_c35eaf4f2426412381b537c1bcb048f7 [id=8439d7bbb943423f9c433fd0c82f1773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.697688 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 (DEFAULT_TABLE table=default.Table_29547e0639d84d1d803dd3f2a9fed320 [id=8c60737562ee4762a3b78a45829449e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.698139 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599f6a7b76d14c9589e5cabb04bb5f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.700359 12822 tablet_service.cc:1467] Processing CreateTablet for tablet b41513bf7aa1480fa95f18c56234790b (DEFAULT_TABLE table=default.Table_1104e7d94ecc44c2917aced82c7f7baa [id=a9c8c600d82f4fcd81f4b42b67fe2cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.700526 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e64cb2f31b64b07848c2dfed18ab9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.700688 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b41513bf7aa1480fa95f18c56234790b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.702481 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 629) took 3336 ms. Trace:
I20240506 19:56:49.703006 13242 rpcz_store.cc:276] 0506 19:56:46.366288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.368783 (+ 2495us) service_pool.cc:225] Handling call
0506 19:56:49.701616 (+3332833us) ts_tablet_manager.cc:1120] Deleting tablet 1750c4dd8cdc40509c553b63d166c466
0506 19:56:49.701627 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.702460 (+ 833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3332403}
W20240506 19:56:49.705394 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1750c4dd8cdc40509c553b63d166c466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1750c4dd8cdc40509c553b63d166c466 already in progress: creating tablet
I20240506 19:56:49.705857 13098 tablet_service.cc:1467] Processing CreateTablet for tablet ad35eed062e843d7a17b3eed22fd90d9 (DEFAULT_TABLE table=default.Table_1458eb9e9352452f88190029f7dfc864 [id=667533b391c748afb927260245b13130]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.706264 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad35eed062e843d7a17b3eed22fd90d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.707458 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 628) took 3341 ms. Trace:
I20240506 19:56:49.707989 13242 rpcz_store.cc:276] 0506 19:56:46.366158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.373530 (+ 7372us) service_pool.cc:225] Handling call
0506 19:56:49.706384 (+3332854us) ts_tablet_manager.cc:1120] Deleting tablet 42151ea2c1214c149d7dcac9a2c447aa
0506 19:56:49.706396 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.707200 (+ 804us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3332353}
W20240506 19:56:49.709600 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 42151ea2c1214c149d7dcac9a2c447aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42151ea2c1214c149d7dcac9a2c447aa already in progress: creating tablet
I20240506 19:56:49.711017 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 632) took 3334 ms. Trace:
I20240506 19:56:49.713078 13090 tablet_service.cc:1467] Processing CreateTablet for tablet fee3e6be377b42818aaa7dda59a1ffda (DEFAULT_TABLE table=default.Table_aa42760da3bc4705bb6912cc7ebbfb74 [id=246f3a1786da4e438d563fa8ead150c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.713541 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3e6be377b42818aaa7dda59a1ffda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.713645 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 599f6a7b76d14c9589e5cabb04bb5f4c (DEFAULT_TABLE table=default.Table_c35eaf4f2426412381b537c1bcb048f7 [id=8439d7bbb943423f9c433fd0c82f1773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.713951 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599f6a7b76d14c9589e5cabb04bb5f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.715551 14069 raft_consensus.cc:482] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.715762 14069 raft_consensus.cc:504] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.716262 14210 raft_consensus.cc:686] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:49.648487 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a1401627948bbaff78fae4d42ab22" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.716540 14210 consensus_queue.cc:237] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.651585 13243 tablet_replica.cc:330] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:49.652201 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 71e4cd127e494acd8b7dffd87cb6bfa0 (DEFAULT_TABLE table=default.Table_36502249055f419692c8fe55e7f7d2ea [id=59360d532f304a3bbab1c482d0ba8c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.652369 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac353c6b9e541f181e20ad72775899a (DEFAULT_TABLE table=default.Table_1258c1a3736349b0973af069d7d58b02 [id=1ab3cd1ac4c744dbb1b0cb55267eea97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.656258 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed6da25f3b94b26959e66e513088c16 (DEFAULT_TABLE table=default.Table_ceed94ec290a471ca0114a9727776f20 [id=54ebe9481ac5420b8adc3c32a9af518a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.717013 13243 raft_consensus.cc:2229] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.717235 13243 raft_consensus.cc:2258] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.717342 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e4cd127e494acd8b7dffd87cb6bfa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.667958 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 (DEFAULT_TABLE table=default.Table_29547e0639d84d1d803dd3f2a9fed320 [id=8c60737562ee4762a3b78a45829449e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.672637 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607b3fc2c17b4793af9b84c19066ec5e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:49.675321 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 28b1e772db304a038a46d428b8c522c4 (DEFAULT_TABLE table=default.Table_112c674a2f6a46cb8e60669eb015488f [id=d50032a383514ade85a7e639245332b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.679694 12962 tablet_service.cc:1467] Processing CreateTablet for tablet fee3e6be377b42818aaa7dda59a1ffda (DEFAULT_TABLE table=default.Table_aa42760da3bc4705bb6912cc7ebbfb74 [id=246f3a1786da4e438d563fa8ead150c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.681447 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8989f02f25418ea9f824b3e5eeddd6 (DEFAULT_TABLE table=default.Table_74493d529d484630900cd65b7b1777da [id=69f26966a0b84d4e9ada2962c959c584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.691792 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 034d6309de0740d0a5806b3847e334cb with delete_type TABLET_DATA_DELETED (replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.692426 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fb091762144f6591152680a6acd29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.692868 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.697078 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 599f6a7b76d14c9589e5cabb04bb5f4c (DEFAULT_TABLE table=default.Table_c35eaf4f2426412381b537c1bcb048f7 [id=8439d7bbb943423f9c433fd0c82f1773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.718147 12982 raft_consensus.cc:3037] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.720284 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6db7d0ad024f53ad76920bcbc0248e (DEFAULT_TABLE table=default.Table_bf2de97d44e34eb08bd8461038259ace [id=44207be9fc564b5bba7a4513e0b3c1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.720986 12982 raft_consensus.cc:2454] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:49.721009 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 7238f1b2905b4b8bb0e2b764ea838c56 (DEFAULT_TABLE table=default.Table_eb33ab4810b94486b70e278e548b02ba [id=bea1ca1264dc4a44a9c2e91d52ae2a81]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:49.721555 13057 leader_election.cc:341] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.721797 13057 leader_election.cc:302] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:49.721987 12819 tablet_service.cc:1467] Processing CreateTablet for tablet c423bb60a6e949c1bed49ffea1b6605a (DEFAULT_TABLE table=default.Table_68a353433c044a28b362eb7d3606bfb6 [id=f5c22b4330f041eea15b7c14968d1732]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.722180 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac353c6b9e541f181e20ad72775899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.722232 12954 tablet_service.cc:1467] Processing CreateTablet for tablet c423bb60a6e949c1bed49ffea1b6605a (DEFAULT_TABLE table=default.Table_68a353433c044a28b362eb7d3606bfb6 [id=f5c22b4330f041eea15b7c14968d1732]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.722497 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c423bb60a6e949c1bed49ffea1b6605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.722592 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c423bb60a6e949c1bed49ffea1b6605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.722756 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed6da25f3b94b26959e66e513088c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.722879 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e64cb2f31b64b07848c2dfed18ab9f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.723099 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b1e772db304a038a46d428b8c522c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.723147 14211 raft_consensus.cc:2781] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.723260 14211 raft_consensus.cc:686] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:49.723484 14211 consensus_queue.cc:237] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.729034 13243 ts_tablet_manager.cc:1875] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.737468 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6db7d0ad024f53ad76920bcbc0248e (DEFAULT_TABLE table=default.Table_bf2de97d44e34eb08bd8461038259ace [id=44207be9fc564b5bba7a4513e0b3c1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.737622 14069 leader_election.cc:288] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.738643 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 7238f1b2905b4b8bb0e2b764ea838c56 (DEFAULT_TABLE table=default.Table_eb33ab4810b94486b70e278e548b02ba [id=bea1ca1264dc4a44a9c2e91d52ae2a81]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:49.739037 12783 leader_election.cc:341] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:49.739544 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6db7d0ad024f53ad76920bcbc0248e (DEFAULT_TABLE table=default.Table_bf2de97d44e34eb08bd8461038259ace [id=44207be9fc564b5bba7a4513e0b3c1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.740218 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6db7d0ad024f53ad76920bcbc0248e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.741003 12781 leader_election.cc:341] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:49.741127 12781 leader_election.cc:302] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.742762 14069 raft_consensus.cc:2726] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.743131 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 7238f1b2905b4b8bb0e2b764ea838c56 (DEFAULT_TABLE table=default.Table_eb33ab4810b94486b70e278e548b02ba [id=bea1ca1264dc4a44a9c2e91d52ae2a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.743743 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7238f1b2905b4b8bb0e2b764ea838c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.748045 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4a6c1aff66ee42a293a7af16d9c5a002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.748306 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607b3fc2c17b4793af9b84c19066ec5e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:49.748755 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a6c1aff66ee42a293a7af16d9c5a002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.749725 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599f6a7b76d14c9589e5cabb04bb5f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.750164 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a1401627948bbaff78fae4d42ab22" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.750190 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_50e680b80e9e48faa1d0e0e07bf41693" } new_table_name: "default.Table_1dabc02b391c4c3386ec45beabfc24ca" modify_external_catalogs: true
I20240506 19:56:49.750376 12922 leader_election.cc:302] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.750563 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607b3fc2c17b4793af9b84c19066ec5e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:49.750797 14143 raft_consensus.cc:2726] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.751281 14143 consensus_queue.cc:1035] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:49.751442 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ceed94ec290a471ca0114a9727776f20" } new_table_name: "default.Table_1a1a7ae174ea4272afca27f6f28b6476" modify_external_catalogs: true
I20240506 19:56:49.752621 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a010c4cc7a5348d79e8c910419d72c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.753135 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a010c4cc7a5348d79e8c910419d72c98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.753551 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee3e6be377b42818aaa7dda59a1ffda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.754295 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5ff6037fa555448c912227cbd9c6280f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.754719 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ff6037fa555448c912227cbd9c6280f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.755743 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1799610d365d46859579977132bedf50" } modify_external_catalogs: true
I20240506 19:56:49.755885 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1799610d365d46859579977132bedf50" } modify_external_catalogs: true
I20240506 19:56:49.757040 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_94fff765fd9146e5a2920c3702f5c8af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.757542 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6db7d0ad024f53ad76920bcbc0248e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.757530 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94fff765fd9146e5a2920c3702f5c8af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.758574 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fb883920a7c94fd4a786a7b48237c418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.758993 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb883920a7c94fd4a786a7b48237c418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.760071 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4607cd1fdeea4e3fa1d76a329c0b0191"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.760509 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4607cd1fdeea4e3fa1d76a329c0b0191 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.761534 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7238f1b2905b4b8bb0e2b764ea838c56. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.761896 13058 leader_election.cc:341] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.762064 13058 leader_election.cc:302] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:49.762589 13058 leader_election.cc:341] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.763028 14222 raft_consensus.cc:2726] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:49.766749 12921 consensus_peers.cc:485] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:49.766829 13095 tablet_service.cc:1467] Processing CreateTablet for tablet c423bb60a6e949c1bed49ffea1b6605a (DEFAULT_TABLE table=default.Table_68a353433c044a28b362eb7d3606bfb6 [id=f5c22b4330f041eea15b7c14968d1732]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.767030 14221 raft_consensus.cc:482] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.767153 14221 raft_consensus.cc:504] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.767505 13094 tablet_service.cc:1467] Processing CreateTablet for tablet d6d9907f476f45d791920f2a60f976a7 (DEFAULT_TABLE table=default.Table_9c5d2e15a9e64b618f68377dce21b541 [id=ad10dbcda08d48c79953b88a4de5eaa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.767868 14221 leader_election.cc:288] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.768450 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_19ad839886424d5baf4207f64dbfbd8b" } new_table_name: "default.Table_eadf31733273444286cc81e91a890ddb" modify_external_catalogs: true
I20240506 19:56:49.768582 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:49.769102 13057 leader_election.cc:341] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.769963 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8989f02f25418ea9f824b3e5eeddd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.697321 12958 tablet_service.cc:1467] Processing CreateTablet for tablet ad35eed062e843d7a17b3eed22fd90d9 (DEFAULT_TABLE table=default.Table_1458eb9e9352452f88190029f7dfc864 [id=667533b391c748afb927260245b13130]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.778491 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad35eed062e843d7a17b3eed22fd90d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.783411 12954 tablet_service.cc:1467] Processing CreateTablet for tablet f3bbf5a4326448d684fcb3f138dcf96d (DEFAULT_TABLE table=default.Table_1799610d365d46859579977132bedf50 [id=1eecc9f214694526b467242d8583397e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.784060 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bbf5a4326448d684fcb3f138dcf96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.782295 13243 ts_tablet_manager.cc:1888] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.785185 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.785367 13243 log.cc:1196] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/4095f9fc28ff48f68064ec8318ff7b51
W20240506 19:56:49.786900 13058 leader_election.cc:341] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.787067 13058 leader_election.cc:302] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.700280 12950 tablet_service.cc:1467] Processing CreateTablet for tablet b41513bf7aa1480fa95f18c56234790b (DEFAULT_TABLE table=default.Table_1104e7d94ecc44c2917aced82c7f7baa [id=a9c8c600d82f4fcd81f4b42b67fe2cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.787809 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b41513bf7aa1480fa95f18c56234790b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.787993 14210 raft_consensus.cc:2726] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.788403 13243 ts_tablet_manager.cc:1909] T 4095f9fc28ff48f68064ec8318ff7b51 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:49.791178 12809 tablet_service.cc:1467] Processing CreateTablet for tablet f3bbf5a4326448d684fcb3f138dcf96d (DEFAULT_TABLE table=default.Table_1799610d365d46859579977132bedf50 [id=1eecc9f214694526b467242d8583397e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.791741 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bbf5a4326448d684fcb3f138dcf96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.645861 13244 tablet_bootstrap.cc:654] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.796569 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 4095f9fc28ff48f68064ec8318ff7b51 (table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]) successfully deleted
I20240506 19:56:49.798264 13243 tablet_replica.cc:330] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:49.798720 13097 tablet_service.cc:1467] Processing CreateTablet for tablet f3bbf5a4326448d684fcb3f138dcf96d (DEFAULT_TABLE table=default.Table_1799610d365d46859579977132bedf50 [id=1eecc9f214694526b467242d8583397e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.799852 14210 raft_consensus.cc:482] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.800107 14210 raft_consensus.cc:504] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.800235 13243 raft_consensus.cc:2229] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.802268 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e50bde3564493eb7267acb42cb8592" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:49.801873 13243 raft_consensus.cc:2258] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:49.802861 13057 leader_election.cc:341] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.803628 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e50bde3564493eb7267acb42cb8592" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.804643 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 63e29e673a784056a79df0b73677f38a (DEFAULT_TABLE table=default.Table_f76cd2f478bd458697b5873a16fbb55b [id=ef23a1148e3648f49b43c73a89b6e598]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.805186 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e29e673a784056a79df0b73677f38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.806483 14210 leader_election.cc:288] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:49.806921 13058 leader_election.cc:341] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.809746 14143 raft_consensus.cc:482] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.809880 14143 raft_consensus.cc:504] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.810482 14143 leader_election.cc:288] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.811131 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fb091762144f6591152680a6acd29e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:49.811704 12921 leader_election.cc:341] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.814019 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 63e29e673a784056a79df0b73677f38a (DEFAULT_TABLE table=default.Table_f76cd2f478bd458697b5873a16fbb55b [id=ef23a1148e3648f49b43c73a89b6e598]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.814570 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e29e673a784056a79df0b73677f38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.814965 13243 ts_tablet_manager.cc:1875] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.807056 13058 leader_election.cc:302] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.817605 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6db7d0ad024f53ad76920bcbc0248e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.818075 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7238f1b2905b4b8bb0e2b764ea838c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.818233 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c423bb60a6e949c1bed49ffea1b6605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.818591 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bbf5a4326448d684fcb3f138dcf96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.818921 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d9907f476f45d791920f2a60f976a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.821933 14210 raft_consensus.cc:2726] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.822001 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fb091762144f6591152680a6acd29e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:49.822223 13123 raft_consensus.cc:2454] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:49.826531 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ed6da25f3b94b26959e66e513088c16 consensus state has no leader
I20240506 19:56:49.828807 13246 ts_tablet_manager.cc:1888] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.828917 13246 log.cc:1196] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2417a535329341d8bb9f8218770f890d
I20240506 19:56:49.828917 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5df015fc4a1f443e816d4e6b097fa6c0" } modify_external_catalogs: true
I20240506 19:56:49.829123 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5df015fc4a1f443e816d4e6b097fa6c0" } modify_external_catalogs: true
I20240506 19:56:49.829208 13246 ts_tablet_manager.cc:1909] T 2417a535329341d8bb9f8218770f890d P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.829211 13243 ts_tablet_manager.cc:1888] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.829519 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 728) took 1768 ms. Trace:
I20240506 19:56:49.829602 13246 rpcz_store.cc:276] 0506 19:56:48.061060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.063063 (+ 2003us) service_pool.cc:225] Handling call
0506 19:56:49.638550 (+1575487us) ts_tablet_manager.cc:1120] Deleting tablet 2417a535329341d8bb9f8218770f890d
0506 19:56:49.638580 (+ 30us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.817553 (+178973us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.828709 (+ 11156us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.829502 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1575352}
I20240506 19:56:49.829905 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_38edce058035412e8151f9b6dae55a46" } new_table_name: "default.Table_07ee733aefe144be9fbd0d5a9cf3dbca" modify_external_catalogs: true
I20240506 19:56:49.830034 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 726) took 1779 ms. Trace:
I20240506 19:56:49.830129 13246 rpcz_store.cc:276] 0506 19:56:48.050942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.065348 (+ 14406us) service_pool.cc:225] Handling call
0506 19:56:49.829888 (+1764540us) ts_tablet_manager.cc:1120] Deleting tablet 8eb9296bb1104b4cbe279eecd3683796
0506 19:56:49.829898 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.829996 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764305}
I20240506 19:56:49.830473 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 725) took 1783 ms. Trace:
I20240506 19:56:49.830554 13246 rpcz_store.cc:276] 0506 19:56:48.047046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.082648 (+ 35602us) service_pool.cc:225] Handling call
0506 19:56:49.830335 (+1747687us) ts_tablet_manager.cc:1120] Deleting tablet cc22261ab2034e818834ba97c2760dad
0506 19:56:49.830343 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.830415 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1747473}
I20240506 19:56:49.830823 13246 tablet_replica.cc:330] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:49.831043 13246 raft_consensus.cc:2229] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.831233 13246 raft_consensus.cc:2258] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.831620 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_014acbfce7444396ad12f76368ca683f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:49.831771 13246 ts_tablet_manager.cc:1875] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:49.832193 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_014acbfce7444396ad12f76368ca683f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:49.833617 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ad7076ed00f4aec9cb92ba7860932b6 consensus state has no leader
I20240506 19:56:49.834044 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a94082868b214538890b435c8c101305"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.834615 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94082868b214538890b435c8c101305 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.835999 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f3bbf5a4326448d684fcb3f138dcf96d
I20240506 19:56:49.834136 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8b6089202064458fa076b29994bd285e" } modify_external_catalogs: true
I20240506 19:56:49.837711 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8b6089202064458fa076b29994bd285e" } modify_external_catalogs: true
I20240506 19:56:49.837754 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bbf5a4326448d684fcb3f138dcf96d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:49.840334 13243 log.cc:1196] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/7de7397db1e74b4488a3f9a70136bccb
I20240506 19:56:49.841894 13243 ts_tablet_manager.cc:1909] T 7de7397db1e74b4488a3f9a70136bccb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:49.843662 13246 ts_tablet_manager.cc:1888] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:49.843812 13246 log.cc:1196] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/03df4632adfa4b0ab999aeaa5bb4e1f4
I20240506 19:56:49.842828 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2417a535329341d8bb9f8218770f890d (table default.Table_c57d5247a9fc4628ac83a4fd916f2b53 [id=0db49a8d38af446f8729cfa002006438]) successfully deleted
W20240506 19:56:49.844401 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8eb9296bb1104b4cbe279eecd3683796 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb9296bb1104b4cbe279eecd3683796 already in progress: creating tablet
W20240506 19:56:49.844657 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet cc22261ab2034e818834ba97c2760dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc22261ab2034e818834ba97c2760dad already in progress: creating tablet
I20240506 19:56:49.844702 13244 tablet_bootstrap.cc:492] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.844835 13244 ts_tablet_manager.cc:1367] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.208s user 0.003s sys 0.000s
I20240506 19:56:49.845883 13246 ts_tablet_manager.cc:1909] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:49.847375 13244 raft_consensus.cc:348] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.847595 13244 raft_consensus.cc:374] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.847671 13244 raft_consensus.cc:729] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.711566 13242 rpcz_store.cc:276] 0506 19:56:46.376334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.376567 (+ 233us) service_pool.cc:225] Handling call
0506 19:56:49.710232 (+3333665us) ts_tablet_manager.cc:1120] Deleting tablet a2e061a5a97646cc9b14ddf741c5b88e
0506 19:56:49.710460 (+ 228us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.710998 (+ 538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3333383}
I20240506 19:56:49.847895 13244 consensus_queue.cc:260] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:49.848292 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a2e061a5a97646cc9b14ddf741c5b88e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e061a5a97646cc9b14ddf741c5b88e already in progress: creating tablet
I20240506 19:56:49.850348 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 7de7397db1e74b4488a3f9a70136bccb (table default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]) successfully deleted
I20240506 19:56:49.851219 13243 tablet_replica.cc:330] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:49.851935 14069 raft_consensus.cc:482] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.852068 14069 raft_consensus.cc:504] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.852430 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 634) took 3462 ms. Trace:
I20240506 19:56:49.852504 13242 rpcz_store.cc:276] 0506 19:56:46.390432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.390488 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:49.852308 (+3461820us) ts_tablet_manager.cc:1120] Deleting tablet cd6245e4adce4038ad6f988877d75045
0506 19:56:49.852317 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.852412 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3461564}
I20240506 19:56:49.852672 14069 leader_election.cc:288] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.852869 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 633) took 3462 ms. Trace:
I20240506 19:56:49.852931 13242 rpcz_store.cc:276] 0506 19:56:46.390266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.392241 (+ 1975us) service_pool.cc:225] Handling call
0506 19:56:49.852753 (+3460512us) ts_tablet_manager.cc:1120] Deleting tablet bb04d06a2b084079ac6357415b124752
0506 19:56:49.852763 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.852853 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3460235}
I20240506 19:56:49.853228 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 636) took 3446 ms. Trace:
I20240506 19:56:49.853327 13242 rpcz_store.cc:276] 0506 19:56:46.406844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.406900 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:49.853130 (+3446230us) ts_tablet_manager.cc:1120] Deleting tablet 449eeb67b602425bb40caa69b6a78a5a
0506 19:56:49.853140 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.853214 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3445956}
I20240506 19:56:49.853565 13242 tablet_replica.cc:330] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:49.853735 13242 raft_consensus.cc:2229] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:49.853775 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bbf5a4326448d684fcb3f138dcf96d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.853886 13242 raft_consensus.cc:2258] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:49.853987 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet cd6245e4adce4038ad6f988877d75045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6245e4adce4038ad6f988877d75045 already in progress: creating tablet
I20240506 19:56:49.853895 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bbf5a4326448d684fcb3f138dcf96d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:49.854266 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bb04d06a2b084079ac6357415b124752 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb04d06a2b084079ac6357415b124752 already in progress: creating tablet
I20240506 19:56:49.854439 13242 ts_tablet_manager.cc:1875] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:49.855214 12781 leader_election.cc:341] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: db13b58d17c94e0f82052f823915d9af
I20240506 19:56:49.855746 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet db13b58d17c94e0f82052f823915d9af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:49.856274 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 63e29e673a784056a79df0b73677f38a (DEFAULT_TABLE table=default.Table_f76cd2f478bd458697b5873a16fbb55b [id=ef23a1148e3648f49b43c73a89b6e598]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.856370 14210 raft_consensus.cc:482] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.856338 13243 raft_consensus.cc:2229] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:49.856669 14210 raft_consensus.cc:504] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:49.857635 12783 leader_election.cc:341] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: db13b58d17c94e0f82052f823915d9af
I20240506 19:56:49.857743 12783 leader_election.cc:302] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:49.857807 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 740) took 1737 ms. Trace:
I20240506 19:56:49.857923 13246 rpcz_store.cc:276] 0506 19:56:48.120090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.177557 (+ 57467us) service_pool.cc:225] Handling call
0506 19:56:49.830741 (+1653184us) ts_tablet_manager.cc:1120] Deleting tablet 03df4632adfa4b0ab999aeaa5bb4e1f4
0506 19:56:49.830750 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.840420 (+ 9670us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.843532 (+ 3112us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:49.857770 (+ 14238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":1652913}
I20240506 19:56:49.858156 14069 raft_consensus.cc:2726] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.858217 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e29e673a784056a79df0b73677f38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.858932 13243 pending_rounds.cc:62] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:49.859282 13243 pending_rounds.cc:68] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744075119693824 op_type: NO_OP noop_request { }
I20240506 19:56:49.859882 13243 raft_consensus.cc:2866] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:49.860422 13243 raft_consensus.cc:2258] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:49.863443 13243 ts_tablet_manager.cc:1875] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:49.864558 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 8689ec0a8e134bceb010053b1f5a343d from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:49.865906 14210 leader_election.cc:288] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.866396 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f7171d06948388bc0fdb3514202ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:49.866925 13057 leader_election.cc:341] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.869145 12922 leader_election.cc:302] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.869725 14143 raft_consensus.cc:2781] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:49.869805 14143 raft_consensus.cc:482] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:49.869872 14143 raft_consensus.cc:3037] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.869941 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f7171d06948388bc0fdb3514202ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.871148 12958 tablet_service.cc:1467] Processing CreateTablet for tablet f6802d9e922a47808c387ded893f9ab6 (DEFAULT_TABLE table=default.Table_6c86bd8e4c954a06a24af7379533208a [id=084f9d59425445c1aad3f9171f2c05d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.871553 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6802d9e922a47808c387ded893f9ab6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:49.871853 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7f36b24ff9848a18e23a599a72ffae2 consensus state has no leader
I20240506 19:56:49.872848 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 98571fa2168644969c851a8edd343a7a (DEFAULT_TABLE table=default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.872860 14143 raft_consensus.cc:504] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.873252 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98571fa2168644969c851a8edd343a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.874025 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fb091762144f6591152680a6acd29e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:49.874603 12921 leader_election.cc:341] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.875161 14143 leader_election.cc:288] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:49.877568 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 98571fa2168644969c851a8edd343a7a (DEFAULT_TABLE table=default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.877609 13096 tablet_service.cc:1467] Processing CreateTablet for tablet f6802d9e922a47808c387ded893f9ab6 (DEFAULT_TABLE table=default.Table_6c86bd8e4c954a06a24af7379533208a [id=084f9d59425445c1aad3f9171f2c05d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.877962 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98571fa2168644969c851a8edd343a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.878005 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6802d9e922a47808c387ded893f9ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.879663 13103 tablet_service.cc:1467] Processing CreateTablet for tablet b41513bf7aa1480fa95f18c56234790b (DEFAULT_TABLE table=default.Table_1104e7d94ecc44c2917aced82c7f7baa [id=a9c8c600d82f4fcd81f4b42b67fe2cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.879984 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b41513bf7aa1480fa95f18c56234790b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.881044 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fb091762144f6591152680a6acd29e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:49.881220 13123 raft_consensus.cc:3037] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:49.882740 13245 tablet_bootstrap.cc:492] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.882859 13245 ts_tablet_manager.cc:1367] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.213s user 0.006s sys 0.003s
I20240506 19:56:49.883605 13123 raft_consensus.cc:2454] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:49.883553 13245 raft_consensus.cc:348] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.883726 13245 raft_consensus.cc:374] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.883774 13245 raft_consensus.cc:729] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.883986 13245 consensus_queue.cc:260] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.884658 13245 ts_tablet_manager.cc:1398] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:49.884932 13245 tablet_bootstrap.cc:492] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.885330 12817 tablet_service.cc:1467] Processing CreateTablet for tablet f6802d9e922a47808c387ded893f9ab6 (DEFAULT_TABLE table=default.Table_6c86bd8e4c954a06a24af7379533208a [id=084f9d59425445c1aad3f9171f2c05d2]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:49.878010 13058 leader_election.cc:341] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.885594 13058 leader_election.cc:302] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.885833 14210 raft_consensus.cc:2726] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.886158 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6802d9e922a47808c387ded893f9ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.887192 13245 tablet_bootstrap.cc:654] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.887144 12922 leader_election.cc:302] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.887508 14143 raft_consensus.cc:2781] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:49.887611 14143 raft_consensus.cc:686] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:49.887825 14143 consensus_queue.cc:237] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.888854 13249 tablet_bootstrap.cc:492] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.888993 13249 ts_tablet_manager.cc:1367] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.257s user 0.006s sys 0.061s
I20240506 19:56:49.889930 13249 raft_consensus.cc:348] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.892663 13249 raft_consensus.cc:374] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.893443 13249 raft_consensus.cc:729] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.893757 13249 consensus_queue.cc:260] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.897629 13244 ts_tablet_manager.cc:1398] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.053s user 0.002s sys 0.001s
I20240506 19:56:49.898238 13249 ts_tablet_manager.cc:1398] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240506 19:56:49.899389 13249 tablet_bootstrap.cc:492] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.900739 13244 tablet_bootstrap.cc:492] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.902660 13244 tablet_bootstrap.cc:654] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.905352 12827 tablet_service.cc:1467] Processing CreateTablet for tablet bd901641d6844fbfba2a2336247a8611 (DEFAULT_TABLE table=default.Table_68b03f1b0b0f468494be25ef614a5b08 [id=825fc426cf02442f9ed682b5af9fda40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.906272 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd901641d6844fbfba2a2336247a8611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.906419 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 6ceb1282a53d49ccb11b1c7239e2b364 (DEFAULT_TABLE table=default.Table_bdd8b0bb2f654920a49117008a662af1 [id=f0c703ff6122447b929a5e49ac7f3f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.905517 12954 tablet_service.cc:1467] Processing CreateTablet for tablet bd901641d6844fbfba2a2336247a8611 (DEFAULT_TABLE table=default.Table_68b03f1b0b0f468494be25ef614a5b08 [id=825fc426cf02442f9ed682b5af9fda40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.906677 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 6ceb1282a53d49ccb11b1c7239e2b364 (DEFAULT_TABLE table=default.Table_bdd8b0bb2f654920a49117008a662af1 [id=f0c703ff6122447b929a5e49ac7f3f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.907140 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ceb1282a53d49ccb11b1c7239e2b364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.907194 13090 tablet_service.cc:1467] Processing CreateTablet for tablet bd901641d6844fbfba2a2336247a8611 (DEFAULT_TABLE table=default.Table_68b03f1b0b0f468494be25ef614a5b08 [id=825fc426cf02442f9ed682b5af9fda40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.906896 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ceb1282a53d49ccb11b1c7239e2b364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.908634 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd901641d6844fbfba2a2336247a8611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.912060 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd901641d6844fbfba2a2336247a8611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.915347 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 3 got 0
I20240506 19:56:49.915961 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:49.916344 13249 tablet_bootstrap.cc:654] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.917937 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:49.923045 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 3 got 0
I20240506 19:56:49.926366 13249 tablet_bootstrap.cc:492] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:49.926527 13249 ts_tablet_manager.cc:1367] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20240506 19:56:49.927304 13249 raft_consensus.cc:348] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.927487 13249 raft_consensus.cc:374] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.927538 13249 raft_consensus.cc:729] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.927752 13249 consensus_queue.cc:260] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.928260 13249 ts_tablet_manager.cc:1398] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:49.928517 13249 tablet_bootstrap.cc:492] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:49.931792 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 6ceb1282a53d49ccb11b1c7239e2b364 (DEFAULT_TABLE table=default.Table_bdd8b0bb2f654920a49117008a662af1 [id=f0c703ff6122447b929a5e49ac7f3f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.932209 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ceb1282a53d49ccb11b1c7239e2b364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.933606 13249 tablet_bootstrap.cc:654] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.935452 14211 raft_consensus.cc:482] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.935683 14211 raft_consensus.cc:504] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.936980 14211 leader_election.cc:288] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.937081 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c81ac92d04ddc93cc52e5c01cbc60" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:49.937629 13057 leader_election.cc:341] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.938138 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_27ea3728f32d484faad6006e1746dad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.938738 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27ea3728f32d484faad6006e1746dad4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.940263 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4343df7e98714672b87cdc89128871a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:49.940969 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4343df7e98714672b87cdc89128871a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:49.941951 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c81ac92d04ddc93cc52e5c01cbc60" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.944038 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_96341728d3094446990de463a07dafd1" } modify_external_catalogs: true
I20240506 19:56:49.944195 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6ef46f8805ee4fdfa0250b1af59ef09d" } modify_external_catalogs: true
I20240506 19:56:49.944583 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3f262965cba54133b5872f383dd0c32c" } new_table_name: "default.Table_a11f31d9b0f44309a0c9d67876079c41" modify_external_catalogs: true
W20240506 19:56:49.945144 13058 leader_election.cc:341] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.945272 13058 leader_election.cc:302] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.945747 14211 raft_consensus.cc:2726] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.958165 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 98571fa2168644969c851a8edd343a7a (DEFAULT_TABLE table=default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:49.965528 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_96341728d3094446990de463a07dafd1" } modify_external_catalogs: true
I20240506 19:56:49.966902 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6ef46f8805ee4fdfa0250b1af59ef09d" } modify_external_catalogs: true
I20240506 19:56:49.967155 13245 tablet_bootstrap.cc:492] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:49.967371 13245 ts_tablet_manager.cc:1367] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.003s
I20240506 19:56:49.968211 13245 raft_consensus.cc:348] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.968482 13245 raft_consensus.cc:374] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.968581 13245 raft_consensus.cc:729] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.969048 13245 consensus_queue.cc:260] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.969748 13245 ts_tablet_manager.cc:1398] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:49.972824 13245 tablet_bootstrap.cc:492] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:49.974412 13245 tablet_bootstrap.cc:654] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:49.976203 13244 tablet_bootstrap.cc:492] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:49.976352 13244 ts_tablet_manager.cc:1367] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20240506 19:56:49.977114 13244 raft_consensus.cc:348] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.977342 13244 raft_consensus.cc:374] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:49.977402 13244 raft_consensus.cc:729] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:49.977625 13244 consensus_queue.cc:260] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:49.978217 14211 raft_consensus.cc:482] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:49.978395 14211 raft_consensus.cc:504] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:49.979452 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899fb38bfed4e02b3708f6abec80e0a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:49.979736 14211 leader_election.cc:288] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:49.979981 13244 ts_tablet_manager.cc:1398] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:49.980357 13244 tablet_bootstrap.cc:492] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:49.980340 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899fb38bfed4e02b3708f6abec80e0a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:49.980991 13057 leader_election.cc:341] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:49.981178 13058 leader_election.cc:341] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:49.981241 13058 leader_election.cc:302] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:49.981712 14211 raft_consensus.cc:2726] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:49.986737 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98571fa2168644969c851a8edd343a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:49.993762 13244 tablet_bootstrap.cc:654] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:50.000020 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb4945e5757c401cacb39037459e2b57 consensus state has no leader
I20240506 19:56:50.002684 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 63f56d2180b1472a8168fb3a63589065 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:50.002728 13244 tablet_bootstrap.cc:492] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.003054 13244 ts_tablet_manager.cc:1367] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20240506 19:56:50.004148 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 46fd9906504e4b95a5b05b01dc3ea1c0
I20240506 19:56:50.004169 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.005043 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8321457717fd4ad28ae1292d9e9a8669
I20240506 19:56:50.006628 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 8321457717fd4ad28ae1292d9e9a8669 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.006632 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.006822 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 8321457717fd4ad28ae1292d9e9a8669 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.006863 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.007480 13244 raft_consensus.cc:348] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.007683 13244 raft_consensus.cc:374] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.007747 13244 raft_consensus.cc:729] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.008761 13244 consensus_queue.cc:260] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.009286 13244 ts_tablet_manager.cc:1398] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240506 19:56:50.010246 13244 tablet_bootstrap.cc:492] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.015872 13244 tablet_bootstrap.cc:654] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.016713 13242 ts_tablet_manager.cc:1888] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.016925 13242 log.cc:1196] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1488328703024ae8983add9e4d6afc8f
I20240506 19:56:50.017489 13242 ts_tablet_manager.cc:1909] T 1488328703024ae8983add9e4d6afc8f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:50.020797 13244 tablet_bootstrap.cc:492] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.020939 13244 ts_tablet_manager.cc:1367] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240506 19:56:50.005261 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.021901 13244 raft_consensus.cc:348] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.022115 13244 raft_consensus.cc:374] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.005734 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 8321457717fd4ad28ae1292d9e9a8669 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.022182 13244 raft_consensus.cc:729] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.022377 13244 consensus_queue.cc:260] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.022920 13244 ts_tablet_manager.cc:1398] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:50.023556 13245 tablet_bootstrap.cc:492] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.023743 13245 ts_tablet_manager.cc:1367] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240506 19:56:50.025015 13244 tablet_bootstrap.cc:492] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.028721 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 3 got 0
I20240506 19:56:50.028786 13244 tablet_bootstrap.cc:654] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.030953 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet bafd3c4706f74e7da36a8f3538760f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.035116 13245 raft_consensus.cc:348] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.035372 13245 raft_consensus.cc:374] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.035430 13245 raft_consensus.cc:729] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.035669 13245 consensus_queue.cc:260] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.036798 12516 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 8778a34ab51a4f7cb969ab533abb4a64
I20240506 19:56:50.036839 13245 ts_tablet_manager.cc:1398] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240506 19:56:50.037179 13245 tablet_bootstrap.cc:492] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:50.037249 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a03f574aec984b83ab596e92a6f323fc consensus state has no leader
I20240506 19:56:50.037660 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 41faccd54f3e49ddb48ea5234f6cdab5
I20240506 19:56:50.038651 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 41faccd54f3e49ddb48ea5234f6cdab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.040196 13245 tablet_bootstrap.cc:654] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.045382 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 41faccd54f3e49ddb48ea5234f6cdab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.047811 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e1222c6575614a1d83d72a3751290c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.048354 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:50.048610 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1222c6575614a1d83d72a3751290c35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.048780 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet f8951bf4873e4382bcd42a52368fb376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.048962 13244 tablet_bootstrap.cc:492] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.049122 13244 ts_tablet_manager.cc:1367] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240506 19:56:50.049911 13244 raft_consensus.cc:348] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.050105 13244 raft_consensus.cc:374] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.050158 13244 raft_consensus.cc:729] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.050360 13244 consensus_queue.cc:260] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.051000 13244 ts_tablet_manager.cc:1398] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:50.051467 13244 tablet_bootstrap.cc:492] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.053205 13244 tablet_bootstrap.cc:654] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.056803 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_23fc1ee2934c454982e8fd45aad54dd1" } new_table_name: "default.Table_238856f9108b4514a3714da3821f7111" modify_external_catalogs: true
I20240506 19:56:50.058265 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9362c95fb2df44f88a0a579b79b4ddd1" } new_table_name: "default.Table_260369fbb74b4623ae5419fdb3370d2a" modify_external_catalogs: true
I20240506 19:56:50.058844 13243 ts_tablet_manager.cc:1888] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:50.059053 13243 log.cc:1196] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/03df4632adfa4b0ab999aeaa5bb4e1f4
I20240506 19:56:50.059669 13243 ts_tablet_manager.cc:1909] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.060393 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9f4ce0d827e5469ba860bae2d23245f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.060983 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f4ce0d827e5469ba860bae2d23245f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.061506 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:50.062006 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1b1ba78c0a194d509cee193f3bec562e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.062495 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b1ba78c0a194d509cee193f3bec562e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.065507 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1d3c02a3c0e64178939ba1d042908501" } modify_external_catalogs: true
I20240506 19:56:50.065873 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 637) took 3651 ms. Trace:
I20240506 19:56:50.066243 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1d3c02a3c0e64178939ba1d042908501" } modify_external_catalogs: true
I20240506 19:56:50.066056 13242 rpcz_store.cc:276] 0506 19:56:46.414739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.414792 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:49.853526 (+3438734us) ts_tablet_manager.cc:1120] Deleting tablet 1488328703024ae8983add9e4d6afc8f
0506 19:56:49.853533 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:49.945336 (+ 91803us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.016197 (+ 70861us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.065818 (+ 49621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3438512}
I20240506 19:56:50.068195 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1488328703024ae8983add9e4d6afc8f (table default.Table_3501606fb8de42f68fd7894fa92e1a95 [id=3cabe6d9486d4b6fa610e1a880a5cdaa]) successfully deleted
I20240506 19:56:50.068926 13242 tablet_replica.cc:330] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:50.069388 13242 raft_consensus.cc:2229] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.072973 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1c043170c0ff4721aeb47b27867693f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.073719 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c043170c0ff4721aeb47b27867693f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.071470 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c09e3bf7046ccba6ed25b08e3460e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.077271 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 41faccd54f3e49ddb48ea5234f6cdab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.079823 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2d5a4569d401426eb881b08e83c66e86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.080420 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d5a4569d401426eb881b08e83c66e86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.082870 14080 raft_consensus.cc:482] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.083014 14080 raft_consensus.cc:504] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.083621 14080 leader_election.cc:288] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.084195 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:50.084719 12921 leader_election.cc:341] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.082895 13242 raft_consensus.cc:2258] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.085335 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:50.085455 14211 raft_consensus.cc:482] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.085553 13120 raft_consensus.cc:2454] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:50.085565 14211 raft_consensus.cc:504] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.085947 14069 raft_consensus.cc:482] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.086086 14069 raft_consensus.cc:504] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.086314 14211 leader_election.cc:288] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:50.086711 14069 leader_election.cc:288] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.087023 12922 leader_election.cc:302] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.087424 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971db6ac24d54efca95695ed1c90d4ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:50.087570 14080 raft_consensus.cc:2781] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:50.087641 14080 raft_consensus.cc:482] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:50.087692 14080 raft_consensus.cc:3037] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.088135 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971db6ac24d54efca95695ed1c90d4ed" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:50.089663 12783 leader_election.cc:341] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d05c5ca7ff864fa98f3494032729e978
I20240506 19:56:50.090389 14080 raft_consensus.cc:504] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:50.091348 13058 leader_election.cc:341] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.091362 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:50.091743 13057 leader_election.cc:341] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.091842 13057 leader_election.cc:302] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:50.092375 12781 leader_election.cc:341] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d05c5ca7ff864fa98f3494032729e978
I20240506 19:56:50.092474 12781 leader_election.cc:302] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.092674 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bf2de97d44e34eb08bd8461038259ace" } new_table_name: "default.Table_ea76ff9f6fb54cac83b450414487d2d3" modify_external_catalogs: true
W20240506 19:56:50.092895 12921 leader_election.cc:341] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.093071 14211 raft_consensus.cc:2726] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.092691 14069 raft_consensus.cc:2726] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.092612 13242 ts_tablet_manager.cc:1875] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.093709 14080 leader_election.cc:288] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.093830 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:50.093999 13120 raft_consensus.cc:3037] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.096740 13120 raft_consensus.cc:2454] T be6e733b6efd4041abf465c7bb788ef5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:50.097169 12922 leader_election.cc:302] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.097575 14080 raft_consensus.cc:2781] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:50.097802 14080 raft_consensus.cc:686] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:50.098093 14080 consensus_queue.cc:237] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.099500 12848 raft_consensus.cc:3037] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.100198 14075 consensus_queue.cc:1035] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:50.108565 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e32a40abc4e74586af79474e4b1675c5 consensus state has no leader
W20240506 19:56:50.109606 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c2eb5dac9d34fb09119868bf5a85803 consensus state has no leader
I20240506 19:56:50.110934 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f078559df48f4d23869ddcb4b3fca9fc
I20240506 19:56:50.111987 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet f078559df48f4d23869ddcb4b3fca9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.112318 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet f078559df48f4d23869ddcb4b3fca9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.113240 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 (table default.Table_052649a7995c41f2a0dde09686115a96 [id=5e3d0dbf38d5451188d26847793c20a1]) successfully deleted
I20240506 19:56:50.116631 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e54543d1544b1c8cff3e890e787337 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.116981 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.118005 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.119132 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet f078559df48f4d23869ddcb4b3fca9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.119889 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:50.125828 13249 tablet_bootstrap.cc:492] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:50.126127 13249 ts_tablet_manager.cc:1367] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.198s user 0.007s sys 0.014s
W20240506 19:56:50.126444 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4c2eb5dac9d34fb09119868bf5a85803 (table default.Table_238856f9108b4514a3714da3821f7111 [id=23e0f6bbcebd465ab11df2c1d9fa4e80], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c2eb5dac9d34fb09119868bf5a85803 consensus state has no leader
I20240506 19:56:50.131213 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:50.133483 13249 raft_consensus.cc:348] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.133817 13249 raft_consensus.cc:374] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.133922 13249 raft_consensus.cc:729] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.134167 13249 consensus_queue.cc:260] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:50.137393 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f63e8e12fd3b453a96413f1fc769fb9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63e8e12fd3b453a96413f1fc769fb9f already in progress: creating tablet
I20240506 19:56:50.137820 13243 tablet_replica.cc:330] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.137543 12521 catalog_manager.cc:5484] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:50.141592 14244 tablet.cc:1722] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:50.141980 13249 ts_tablet_manager.cc:1398] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240506 19:56:50.143173 13249 tablet_bootstrap.cc:492] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:50.152912 14211 raft_consensus.cc:482] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.153115 14211 raft_consensus.cc:504] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.153908 14211 leader_election.cc:288] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:50.154376 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8951bf4873e4382bcd42a52368fb376" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:50.155030 13058 leader_election.cc:341] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:50.153599 12921 consensus_peers.cc:485] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.158178 13244 tablet_bootstrap.cc:492] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.158341 13244 ts_tablet_manager.cc:1367] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.002s
I20240506 19:56:50.159206 13244 raft_consensus.cc:348] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.159443 13244 raft_consensus.cc:374] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.159507 13244 raft_consensus.cc:729] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.159736 13244 consensus_queue.cc:260] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.160146 13243 raft_consensus.cc:2229] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:50.160298 13244 ts_tablet_manager.cc:1398] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:50.161954 14243 tablet.cc:1722] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:50.162530 13243 raft_consensus.cc:2258] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.164748 13243 ts_tablet_manager.cc:1875] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.167261 13249 tablet_bootstrap.cc:654] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.177618 13244 tablet_bootstrap.cc:492] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.178526 13249 tablet_bootstrap.cc:492] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:50.179311 13244 tablet_bootstrap.cc:654] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.179555 13249 ts_tablet_manager.cc:1367] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.002s
I20240506 19:56:50.181046 13245 tablet_bootstrap.cc:492] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.181888 14069 raft_consensus.cc:482] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.182035 14069 raft_consensus.cc:504] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.182178 13249 raft_consensus.cc:348] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.182358 13249 raft_consensus.cc:374] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.182405 13249 raft_consensus.cc:729] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.182682 14069 leader_election.cc:288] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.182586 13249 consensus_queue.cc:260] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:50.185570 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a6db7d0ad024f53ad76920bcbc0248e consensus state has no leader
W20240506 19:56:50.186950 12783 leader_election.cc:341] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: e17197adc6ca4593ac3fc494548a6e67
I20240506 19:56:50.188102 13245 ts_tablet_manager.cc:1367] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.006s
I20240506 19:56:50.188875 13245 raft_consensus.cc:348] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.189215 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet e17197adc6ca4593ac3fc494548a6e67 with delete_type TABLET_DATA_DELETED (replaced by tablet 4095f9fc28ff48f68064ec8318ff7b51) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.189500 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:50.189100 13245 raft_consensus.cc:374] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.191514 13245 raft_consensus.cc:729] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.191797 13245 consensus_queue.cc:260] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.197693 13245 ts_tablet_manager.cc:1398] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:50.204622 13245 tablet_bootstrap.cc:492] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:50.206044 13245 tablet_bootstrap.cc:654] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.209884 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a11f31d9b0f44309a0c9d67876079c41" } new_table_name: "default.Table_47913566bb1846d19fddafa8a3a65bd6" modify_external_catalogs: true
I20240506 19:56:50.212985 13246 tablet_replica.cc:330] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:50.213240 13246 raft_consensus.cc:2229] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.213418 13246 raft_consensus.cc:2258] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.214061 13246 ts_tablet_manager.cc:1875] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.214336 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 (table default.Table_052649a7995c41f2a0dde09686115a96 [id=5e3d0dbf38d5451188d26847793c20a1]) successfully deleted
I20240506 19:56:50.216563 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_eebcdd01355340739cb45d1acca7daa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.217176 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eebcdd01355340739cb45d1acca7daa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.227538 13244 tablet_bootstrap.cc:492] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.227705 13244 ts_tablet_manager.cc:1367] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.001s
I20240506 19:56:50.228571 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_50e723eab6834893b76afae99961fb99" } modify_external_catalogs: true
I20240506 19:56:50.228535 13244 raft_consensus.cc:348] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.228766 13244 raft_consensus.cc:374] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.228819 13244 raft_consensus.cc:729] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.228804 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_50e723eab6834893b76afae99961fb99" } modify_external_catalogs: true
I20240506 19:56:50.229436 12514 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b] alter complete (version 3)
I20240506 19:56:50.231575 13242 ts_tablet_manager.cc:1888] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.232234 13242 log.cc:1196] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/618c56ef126a4ca48b6a379a6771bc1c
I20240506 19:56:50.234740 13242 ts_tablet_manager.cc:1909] T 618c56ef126a4ca48b6a379a6771bc1c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:50.235687 13244 consensus_queue.cc:260] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.236454 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_de4daf2bb6824e22ab5ee4ae55a493d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.237130 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de4daf2bb6824e22ab5ee4ae55a493d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.240295 12515 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:50.242743 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acde41b2c03440a89243aae008779e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.244908 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 1372be71e1d141ed89e58cdb806431a4 with delete_type TABLET_DATA_DELETED (replaced by tablet a1160ab972124630a9be7abeefb4aa56) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.244109 13244 ts_tablet_manager.cc:1398] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240506 19:56:50.245682 13244 tablet_bootstrap.cc:492] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.250771 12515 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
W20240506 19:56:50.254148 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a03f574aec984b83ab596e92a6f323fc consensus state has no leader
I20240506 19:56:50.254284 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 650) took 3542 ms. Trace:
I20240506 19:56:50.256139 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea8710b001ee40d9aec404921fb4dcd3" } modify_external_catalogs: true
I20240506 19:56:50.256495 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea8710b001ee40d9aec404921fb4dcd3" } modify_external_catalogs: true
I20240506 19:56:50.257534 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_94fff765fd9146e5a2920c3702f5c8af" } modify_external_catalogs: true
I20240506 19:56:50.257746 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_94fff765fd9146e5a2920c3702f5c8af" } modify_external_catalogs: true
I20240506 19:56:50.255797 13242 rpcz_store.cc:276] 0506 19:56:46.711319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.711393 (+ 74us) service_pool.cc:225] Handling call
0506 19:56:50.068817 (+3357424us) ts_tablet_manager.cc:1120] Deleting tablet 618c56ef126a4ca48b6a379a6771bc1c
0506 19:56:50.068830 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.141261 (+ 72431us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.230972 (+ 89711us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.253958 (+ 22986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3357107}
I20240506 19:56:50.258468 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f83a7797a045469bbc2a87de8b086a2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.259222 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 618c56ef126a4ca48b6a379a6771bc1c (table default.Table_b50238125ad14920bf820245406918c3 [id=7f61579ec04443a6992e572f3e52f980]) successfully deleted
W20240506 19:56:50.259735 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f83a7797a045469bbc2a87de8b086a2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.260954 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 652) took 3545 ms. Trace:
I20240506 19:56:50.262321 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_29547e0639d84d1d803dd3f2a9fed320" } modify_external_catalogs: true
I20240506 19:56:50.262537 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_29547e0639d84d1d803dd3f2a9fed320" } modify_external_catalogs: true
I20240506 19:56:50.261937 13242 rpcz_store.cc:276] 0506 19:56:46.715282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.715348 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:50.259610 (+3544262us) ts_tablet_manager.cc:1120] Deleting tablet 9227f74e85914cafabad913dc6dfba04
0506 19:56:50.259913 (+ 303us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.260930 (+ 1017us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3543936}
I20240506 19:56:50.263582 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7b12e2d9a5d0496586f34ba591615326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.264160 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b12e2d9a5d0496586f34ba591615326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.265743 13244 tablet_bootstrap.cc:654] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.266242 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 651) took 3551 ms. Trace:
I20240506 19:56:50.268548 13246 ts_tablet_manager.cc:1888] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.268677 13246 log.cc:1196] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/30ad527c3f2a4b53affa928e149db1df
I20240506 19:56:50.268303 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_351b532454104aebb7decad2864d0629" } new_table_name: "default.Table_1d4371bd92c34570ba9ffca5ae2a387d" modify_external_catalogs: true
I20240506 19:56:50.268975 13246 ts_tablet_manager.cc:1909] T 30ad527c3f2a4b53affa928e149db1df P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:50.269971 13242 rpcz_store.cc:276] 0506 19:56:46.715121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.715715 (+ 594us) service_pool.cc:225] Handling call
0506 19:56:50.264656 (+3548941us) ts_tablet_manager.cc:1120] Deleting tablet 9fed2f12eda54be0879b1e9c33ca3e31
0506 19:56:50.264962 (+ 306us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.266218 (+ 1256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3548386}
I20240506 19:56:50.273356 13244 tablet_bootstrap.cc:492] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.273527 13244 ts_tablet_manager.cc:1367] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20240506 19:56:50.274353 13244 raft_consensus.cc:348] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.274590 13244 raft_consensus.cc:374] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.274664 13244 raft_consensus.cc:729] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.274909 13244 consensus_queue.cc:260] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.275455 13244 ts_tablet_manager.cc:1398] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:50.275962 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 653) took 3476 ms. Trace:
I20240506 19:56:50.276100 13242 rpcz_store.cc:276] 0506 19:56:46.799875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:46.805323 (+ 5448us) service_pool.cc:225] Handling call
0506 19:56:50.273566 (+3468243us) ts_tablet_manager.cc:1120] Deleting tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa
0506 19:56:50.275782 (+ 2216us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.275940 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3467499}
I20240506 19:56:50.280112 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 739) took 2160 ms. Trace:
I20240506 19:56:50.280236 13246 rpcz_store.cc:276] 0506 19:56:48.119986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.183777 (+ 63791us) service_pool.cc:225] Handling call
0506 19:56:50.212917 (+2029140us) ts_tablet_manager.cc:1120] Deleting tablet 30ad527c3f2a4b53affa928e149db1df
0506 19:56:50.212930 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.218100 (+ 5170us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.268424 (+ 50324us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.280080 (+ 11656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2028938}
W20240506 19:56:50.265570 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9227f74e85914cafabad913dc6dfba04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9227f74e85914cafabad913dc6dfba04 already in progress: creating tablet
I20240506 19:56:50.281087 13242 tablet_replica.cc:330] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:50.281072 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9fed2f12eda54be0879b1e9c33ca3e31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fed2f12eda54be0879b1e9c33ca3e31 already in progress: creating tablet
W20240506 19:56:50.281366 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f30a2ce53464d12b98c6a0c2fdeb4fa already in progress: creating tablet
I20240506 19:56:50.281271 13242 raft_consensus.cc:2229] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.281522 13242 raft_consensus.cc:2258] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.282895 13244 tablet_bootstrap.cc:492] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.285632 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 30ad527c3f2a4b53affa928e149db1df (table default.Table_052f7d1eab54464d8743bd859c9f5bad [id=cdb5ce64abc74bb3bf60d7aa727c515f]) successfully deleted
I20240506 19:56:50.286324 13242 ts_tablet_manager.cc:1875] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:50.194885 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 49892.1us
I20240506 19:56:50.248158 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:50.295405 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8951bf4873e4382bcd42a52368fb376" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:50.295641 12984 raft_consensus.cc:2454] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:50.295614 12781 leader_election.cc:341] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: e17197adc6ca4593ac3fc494548a6e67
I20240506 19:56:50.295743 12781 leader_election.cc:302] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.296005 14069 raft_consensus.cc:2726] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.297271 13057 leader_election.cc:302] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.297402 14250 consensus_queue.cc:1035] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:50.297701 14211 raft_consensus.cc:2781] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:50.297770 14211 raft_consensus.cc:482] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:50.249516 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7f7171d06948388bc0fdb3514202ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.297822 14211 raft_consensus.cc:3037] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.300427 14211 raft_consensus.cc:504] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.301054 14211 leader_election.cc:288] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:50.302878 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8951bf4873e4382bcd42a52368fb376" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:50.303481 13058 leader_election.cc:341] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.304036 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8951bf4873e4382bcd42a52368fb376" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:50.304224 12984 raft_consensus.cc:3037] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.305231 13249 ts_tablet_manager.cc:1398] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.124s user 0.001s sys 0.001s
I20240506 19:56:50.306324 13249 tablet_bootstrap.cc:492] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:50.307309 13244 tablet_bootstrap.cc:654] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:50.309748 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c882759dc0f4ba58aafaab3b855fa30 consensus state has no leader
I20240506 19:56:50.313557 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03fcaa05d04ebd9f262f43ff7c02e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f7984f6e0fe485fa0c15fd13c3f428e) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.306998 12984 raft_consensus.cc:2454] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:50.315779 13057 leader_election.cc:302] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.316344 14211 raft_consensus.cc:2781] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:50.316478 14211 raft_consensus.cc:686] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:50.316717 14211 consensus_queue.cc:237] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.321486 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b9c18afe9e154517909885ea89fefcc2" } new_table_name: "default.Table_e4fb89d4dd964179b1cd1c2f2c91567c" modify_external_catalogs: true
I20240506 19:56:50.324532 13245 tablet_bootstrap.cc:492] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.324846 13245 ts_tablet_manager.cc:1367] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.120s user 0.007s sys 0.007s
I20240506 19:56:50.325601 13245 raft_consensus.cc:348] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.325812 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c39e942c7b1948ce8b2d1e47289bcbd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.326189 13245 raft_consensus.cc:374] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.326337 13245 raft_consensus.cc:729] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:50.326591 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c39e942c7b1948ce8b2d1e47289bcbd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.326812 13245 consensus_queue.cc:260] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.327812 13245 ts_tablet_manager.cc:1398] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 19:56:50.327840 14069 raft_consensus.cc:482] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.327991 14069 raft_consensus.cc:504] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.328500 13245 tablet_bootstrap.cc:492] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:50.328609 14069 leader_election.cc:288] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:50.329609 12783 leader_election.cc:341] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ef03fcaa05d04ebd9f262f43ff7c02e4
W20240506 19:56:50.330633 12781 leader_election.cc:341] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:50.330732 12781 leader_election.cc:302] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.331200 14069 raft_consensus.cc:2726] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.335098 13249 tablet_bootstrap.cc:654] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.335901 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_912ad4a3913f41e394764648805c3a30" } new_table_name: "default.Table_8d7fd547dfb74b66a34d4207ef4bf800" modify_external_catalogs: true
I20240506 19:56:50.336308 14069 raft_consensus.cc:482] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.336439 14069 raft_consensus.cc:504] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.337076 14069 leader_election.cc:288] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:50.337985 12783 leader_election.cc:341] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 517c09e3bf7046ccba6ed25b08e3460e
W20240506 19:56:50.339005 12781 leader_election.cc:341] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 517c09e3bf7046ccba6ed25b08e3460e
I20240506 19:56:50.339105 12781 leader_election.cc:302] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.339490 14069 raft_consensus.cc:2726] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.342185 13245 tablet_bootstrap.cc:654] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.343994 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f00049dcfdce4881891eed1bf5951d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.344573 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f00049dcfdce4881891eed1bf5951d5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.344875 13244 tablet_bootstrap.cc:492] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.345003 13244 ts_tablet_manager.cc:1367] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240506 19:56:50.347604 13244 raft_consensus.cc:348] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.347798 13244 raft_consensus.cc:374] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.347859 13244 raft_consensus.cc:729] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.348067 13244 consensus_queue.cc:260] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.355003 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_07942e78a30c4f24b8a56a39c0f39e4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.355713 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07942e78a30c4f24b8a56a39c0f39e4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:50.340734 12921 consensus_peers.cc:485] T b35231d3f1bb4348aa7af3c71b809cd9 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.361647 13243 ts_tablet_manager.cc:1888] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:50.361790 13243 log.cc:1196] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/aae313961a914ed485f866d85785a788
I20240506 19:56:50.366279 13244 ts_tablet_manager.cc:1398] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240506 19:56:50.367159 13246 tablet_replica.cc:330] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:50.367367 13246 raft_consensus.cc:2229] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.367522 13246 raft_consensus.cc:2258] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.368024 13246 ts_tablet_manager.cc:1875] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.368264 13244 tablet_bootstrap.cc:492] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.369884 13244 tablet_bootstrap.cc:654] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:50.372385 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df160fd5d8ce4bc192913ce702f8f478 consensus state has no leader
I20240506 19:56:50.375020 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0b1a8466d287416ab4ec1f38a4687e69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.375614 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b1a8466d287416ab4ec1f38a4687e69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.377453 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_238856f9108b4514a3714da3821f7111" } modify_external_catalogs: true
I20240506 19:56:50.377657 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_238856f9108b4514a3714da3821f7111" } modify_external_catalogs: true
W20240506 19:56:50.378803 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d09e7b64b6d45618c6595c1013922df consensus state has no leader
I20240506 19:56:50.379067 12517 master_service.cc:946] ReplaceTablet: received request to replace tablet d6edff8e7e6945aa926aa1486cbbc2bc from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:50.388621 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:50.391359 13243 ts_tablet_manager.cc:1909] T aae313961a914ed485f866d85785a788 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.391546 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:50.398397 13057 consensus_peers.cc:485] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7de7397db1e74b4488a3f9a70136bccb. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.400478 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_c9f790860bb44630a84fb411e08d50d5 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 3 got 0
I20240506 19:56:50.402446 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 803) took 1031 ms. Trace:
I20240506 19:56:50.403326 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e3ab58eefbbc4d9fac784342d0775037" } new_table_name: "default.Table_f44e310a873e4dd28f57ab5ff4e28050" modify_external_catalogs: true
I20240506 19:56:50.409323 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_113b557a89234f5696e404ae3a3cc0e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.409920 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_113b557a89234f5696e404ae3a3cc0e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.414336 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1f0e4245774ccab0aa9c6c2ba49c75 (DEFAULT_TABLE table=default.Table_5ff6037fa555448c912227cbd9c6280f [id=42d9312c57544c3cad708124ec023b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.414769 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1f0e4245774ccab0aa9c6c2ba49c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.414979 12819 tablet_service.cc:1467] Processing CreateTablet for tablet cd004194b60d42579f57ea0c6d4c8760 (DEFAULT_TABLE table=default.Table_0e783c507b7342758e013828bcca3d66 [id=6aa9f4187bbf422dbff68fefa9a4d13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.415421 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd004194b60d42579f57ea0c6d4c8760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.416851 13243 rpcz_store.cc:276] 0506 19:56:49.371220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.371551 (+ 331us) service_pool.cc:225] Handling call
0506 19:56:50.137732 (+766181us) ts_tablet_manager.cc:1120] Deleting tablet aae313961a914ed485f866d85785a788
0506 19:56:50.137740 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.304034 (+166294us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.361523 (+ 57489us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.402421 (+ 40898us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22048,"tablet-delete.queue_time_us":765978}
I20240506 19:56:50.413033 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1f0e4245774ccab0aa9c6c2ba49c75 (DEFAULT_TABLE table=default.Table_5ff6037fa555448c912227cbd9c6280f [id=42d9312c57544c3cad708124ec023b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.417588 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1f0e4245774ccab0aa9c6c2ba49c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.417594 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 141bca5c816a494b83704a915f0edfd1 (DEFAULT_TABLE table=default.Table_946dd68492dd4528a7925366e34068ba [id=2a21ffaa40e34bf1a4efd20e3abe61f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.417769 13243 tablet_replica.cc:330] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.417976 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141bca5c816a494b83704a915f0edfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.418359 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1f0e4245774ccab0aa9c6c2ba49c75 (DEFAULT_TABLE table=default.Table_5ff6037fa555448c912227cbd9c6280f [id=42d9312c57544c3cad708124ec023b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.418630 13243 raft_consensus.cc:2229] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:50.418715 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1f0e4245774ccab0aa9c6c2ba49c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.418972 13243 raft_consensus.cc:2258] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.419504 13243 ts_tablet_manager.cc:1875] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.413205 12954 tablet_service.cc:1467] Processing CreateTablet for tablet cd004194b60d42579f57ea0c6d4c8760 (DEFAULT_TABLE table=default.Table_0e783c507b7342758e013828bcca3d66 [id=6aa9f4187bbf422dbff68fefa9a4d13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.421260 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 23db7118b4914afa95eaed713cca8839 (DEFAULT_TABLE table=default.Table_59e7933756ae440cb5377ffc3ca93c29 [id=c68d44dcff8e4c528981ab7206ee4dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.421480 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd004194b60d42579f57ea0c6d4c8760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.421592 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23db7118b4914afa95eaed713cca8839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.421556 13090 tablet_service.cc:1467] Processing CreateTablet for tablet cd004194b60d42579f57ea0c6d4c8760 (DEFAULT_TABLE table=default.Table_0e783c507b7342758e013828bcca3d66 [id=6aa9f4187bbf422dbff68fefa9a4d13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.421902 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd004194b60d42579f57ea0c6d4c8760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.413522 12952 tablet_service.cc:1467] Processing CreateTablet for tablet e426f9625b19426e8bab707f967ba19e (DEFAULT_TABLE table=default.Table_49169eeefc1442ce942d129a96fc7e55 [id=d0387abfe73c4e16a35a7feef5a12146]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.425567 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e426f9625b19426e8bab707f967ba19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.425566 13103 tablet_service.cc:1467] Processing CreateTablet for tablet e426f9625b19426e8bab707f967ba19e (DEFAULT_TABLE table=default.Table_49169eeefc1442ce942d129a96fc7e55 [id=d0387abfe73c4e16a35a7feef5a12146]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.425904 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e426f9625b19426e8bab707f967ba19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.426072 12817 tablet_service.cc:1467] Processing CreateTablet for tablet e426f9625b19426e8bab707f967ba19e (DEFAULT_TABLE table=default.Table_49169eeefc1442ce942d129a96fc7e55 [id=d0387abfe73c4e16a35a7feef5a12146]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.426385 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e426f9625b19426e8bab707f967ba19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.426935 13243 ts_tablet_manager.cc:1888] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:50.427023 13243 log.cc:1196] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f4e4be565e2d4c59b3587baed72e4a9c
I20240506 19:56:50.427326 13243 ts_tablet_manager.cc:1909] T f4e4be565e2d4c59b3587baed72e4a9c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.428088 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 804) took 1001 ms. Trace:
I20240506 19:56:50.428170 13243 rpcz_store.cc:276] 0506 19:56:49.426169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.426231 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:50.417723 (+991492us) ts_tablet_manager.cc:1120] Deleting tablet f4e4be565e2d4c59b3587baed72e4a9c
0506 19:56:50.417732 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.423950 (+ 6218us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.426858 (+ 2908us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.428064 (+ 1206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":991318}
I20240506 19:56:50.414666 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 141bca5c816a494b83704a915f0edfd1 (DEFAULT_TABLE table=default.Table_946dd68492dd4528a7925366e34068ba [id=2a21ffaa40e34bf1a4efd20e3abe61f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.429590 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141bca5c816a494b83704a915f0edfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.429616 13243 tablet_replica.cc:330] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.429602 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 141bca5c816a494b83704a915f0edfd1 (DEFAULT_TABLE table=default.Table_946dd68492dd4528a7925366e34068ba [id=2a21ffaa40e34bf1a4efd20e3abe61f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.429786 13243 raft_consensus.cc:2229] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:50.429930 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141bca5c816a494b83704a915f0edfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.430020 13243 pending_rounds.cc:62] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:50.430063 13243 pending_rounds.cc:68] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744078903029760 op_type: NO_OP noop_request { }
I20240506 19:56:50.430227 13243 raft_consensus.cc:2866] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:50.430302 13243 raft_consensus.cc:2258] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.432723 14069 raft_consensus.cc:482] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.432863 14069 raft_consensus.cc:504] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.414973 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 23db7118b4914afa95eaed713cca8839 (DEFAULT_TABLE table=default.Table_59e7933756ae440cb5377ffc3ca93c29 [id=c68d44dcff8e4c528981ab7206ee4dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.433562 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23db7118b4914afa95eaed713cca8839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.433543 14069 leader_election.cc:288] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:50.434958 12783 leader_election.cc:341] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 6d04a1c5ee7b4d529fa2f4da4945ff09
W20240506 19:56:50.437969 12781 leader_election.cc:341] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:50.438084 12781 leader_election.cc:302] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.438452 13243 ts_tablet_manager.cc:1875] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.438568 14069 raft_consensus.cc:2726] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.442929 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 23db7118b4914afa95eaed713cca8839 (DEFAULT_TABLE table=default.Table_59e7933756ae440cb5377ffc3ca93c29 [id=c68d44dcff8e4c528981ab7206ee4dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.443512 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23db7118b4914afa95eaed713cca8839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.444093 13243 ts_tablet_manager.cc:1888] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:50.444211 13243 log.cc:1196] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/17fb091762144f6591152680a6acd29e
I20240506 19:56:50.444375 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]) successfully deleted
I20240506 19:56:50.444586 13243 ts_tablet_manager.cc:1909] T 17fb091762144f6591152680a6acd29e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.445164 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f4e4be565e2d4c59b3587baed72e4a9c (table default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]) successfully deleted
W20240506 19:56:50.445521 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 3fb398132c58489fb0a28492302e4a23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb398132c58489fb0a28492302e4a23 already in progress: creating tablet
I20240506 19:56:50.445750 13243 tablet_replica.cc:330] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.445952 13243 raft_consensus.cc:2229] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.446210 12823 tablet_service.cc:1467] Processing CreateTablet for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 (DEFAULT_TABLE table=default.Table_cd37c88af8ee458ea4d03e8901f98d9f [id=cc1abddccb024092b63578c266f7ff9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.447293 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4cdb7dd4e44b45963cf93eaefe23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.446106 13243 raft_consensus.cc:2258] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.447837 13243 ts_tablet_manager.cc:1875] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:50.452127 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 58c88f80f5b34672b7c3a4b604469bf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58c88f80f5b34672b7c3a4b604469bf1 already in progress: creating tablet
I20240506 19:56:50.453758 13243 ts_tablet_manager.cc:1888] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.453876 13243 log.cc:1196] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/607b3fc2c17b4793af9b84c19066ec5e
I20240506 19:56:50.454154 13243 ts_tablet_manager.cc:1909] T 607b3fc2c17b4793af9b84c19066ec5e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.455193 13243 tablet_replica.cc:330] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.455403 13243 raft_consensus.cc:2229] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.455541 13243 raft_consensus.cc:2258] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.456043 13090 tablet_service.cc:1467] Processing CreateTablet for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 (DEFAULT_TABLE table=default.Table_cd37c88af8ee458ea4d03e8901f98d9f [id=cc1abddccb024092b63578c266f7ff9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.456444 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4cdb7dd4e44b45963cf93eaefe23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.459817 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_204da63924eb46ad9044cbc009296da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.460458 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_204da63924eb46ad9044cbc009296da3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.461325 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 76fb4d67510e413d9f23b021dae43b0f (DEFAULT_TABLE table=default.Table_aa2cfdcd14e74072be6f657fcb05127e [id=a5c3329d55274c708fa24ff378b7cea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.461731 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb4d67510e413d9f23b021dae43b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.462792 13096 tablet_service.cc:1467] Processing CreateTablet for tablet ee8a853d11844ccea1ca4e1d2f43dd2a (DEFAULT_TABLE table=default.Table_be02490c0e0c43c29452e03dd1d15be0 [id=66efec0f95ae4e269ef46d7c9ddbe0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.463147 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8a853d11844ccea1ca4e1d2f43dd2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.464601 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 979d789bc89a435b9e6cd8d6dc9c2037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 979d789bc89a435b9e6cd8d6dc9c2037 already in progress: creating tablet
I20240506 19:56:50.464862 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 76fb4d67510e413d9f23b021dae43b0f (DEFAULT_TABLE table=default.Table_aa2cfdcd14e74072be6f657fcb05127e [id=a5c3329d55274c708fa24ff378b7cea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.465315 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb4d67510e413d9f23b021dae43b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.465596 12813 tablet_service.cc:1467] Processing CreateTablet for tablet ee8a853d11844ccea1ca4e1d2f43dd2a (DEFAULT_TABLE table=default.Table_be02490c0e0c43c29452e03dd1d15be0 [id=66efec0f95ae4e269ef46d7c9ddbe0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.465974 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8a853d11844ccea1ca4e1d2f43dd2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.468852 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 744864043f9347e0a13f8c2a7d6b147f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 744864043f9347e0a13f8c2a7d6b147f already in progress: creating tablet
I20240506 19:56:50.468936 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 73c4aba81db7470f87728def61bbf71c (DEFAULT_TABLE table=default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6 [id=0090970ea0b54ee4a872766d8145bd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.469240 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 17fb091762144f6591152680a6acd29e (table default.Table_82d6bee639334fca838d2adf031efd21 [id=d8e5bdbac7204b04ba040114b49d6ee8]) successfully deleted
I20240506 19:56:50.469332 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c4aba81db7470f87728def61bbf71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.469640 13249 tablet_bootstrap.cc:492] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:50.469745 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 607b3fc2c17b4793af9b84c19066ec5e (table default.Table_c3e795b5a9724754a33f79ebe9ac2c1f [id=343720fdaa3d43b28031433246aebe9c]) successfully deleted
I20240506 19:56:50.469805 13249 ts_tablet_manager.cc:1367] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.164s user 0.000s sys 0.003s
I20240506 19:56:50.470548 13249 raft_consensus.cc:348] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.470773 13249 raft_consensus.cc:374] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.470845 13249 raft_consensus.cc:729] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.471051 13249 consensus_queue.cc:260] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.471571 13249 ts_tablet_manager.cc:1398] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:50.471876 13249 tablet_bootstrap.cc:492] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:50.472409 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c9f790860bb44630a84fb411e08d50d5" } new_table_name: "default.Table_ecb75e9b5c5d4c4ebd6f3684a6a350f9" modify_external_catalogs: true
I20240506 19:56:50.473074 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 73c4aba81db7470f87728def61bbf71c (DEFAULT_TABLE table=default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6 [id=0090970ea0b54ee4a872766d8145bd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.473582 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c4aba81db7470f87728def61bbf71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.477946 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 76fb4d67510e413d9f23b021dae43b0f (DEFAULT_TABLE table=default.Table_aa2cfdcd14e74072be6f657fcb05127e [id=a5c3329d55274c708fa24ff378b7cea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.478439 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb4d67510e413d9f23b021dae43b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.480090 13249 tablet_bootstrap.cc:654] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.482144 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 4bed507b4ac14edb9f4172c0d75a5fd7 (DEFAULT_TABLE table=default.Table_4a6c1aff66ee42a293a7af16d9c5a002 [id=8e8f6ceb668d4e14a5ad41373ab56a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.482555 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bed507b4ac14edb9f4172c0d75a5fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.486526 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 4bed507b4ac14edb9f4172c0d75a5fd7 (DEFAULT_TABLE table=default.Table_4a6c1aff66ee42a293a7af16d9c5a002 [id=8e8f6ceb668d4e14a5ad41373ab56a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.486943 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bed507b4ac14edb9f4172c0d75a5fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.488998 13243 ts_tablet_manager.cc:1875] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.490593 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 73c4aba81db7470f87728def61bbf71c (DEFAULT_TABLE table=default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6 [id=0090970ea0b54ee4a872766d8145bd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.491079 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c4aba81db7470f87728def61bbf71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.494678 12954 tablet_service.cc:1467] Processing CreateTablet for tablet ee8a853d11844ccea1ca4e1d2f43dd2a (DEFAULT_TABLE table=default.Table_be02490c0e0c43c29452e03dd1d15be0 [id=66efec0f95ae4e269ef46d7c9ddbe0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.495093 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8a853d11844ccea1ca4e1d2f43dd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.495810 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 4bed507b4ac14edb9f4172c0d75a5fd7 (DEFAULT_TABLE table=default.Table_4a6c1aff66ee42a293a7af16d9c5a002 [id=8e8f6ceb668d4e14a5ad41373ab56a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.496289 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bed507b4ac14edb9f4172c0d75a5fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.496774 12809 tablet_service.cc:1467] Processing CreateTablet for tablet e0fbbc77c0b64c2fa3d125e074eecd87 (DEFAULT_TABLE table=default.Table_807ff2b24da64f7396c8dc15bdc46c88 [id=38c0ff2e5c6d457884dbddc850f4c7d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.497143 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fbbc77c0b64c2fa3d125e074eecd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.499648 13242 ts_tablet_manager.cc:1888] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.499760 13242 log.cc:1196] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a36b4d51b100486c831a41a52d81fd66
I20240506 19:56:50.500037 13242 ts_tablet_manager.cc:1909] T a36b4d51b100486c831a41a52d81fd66 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:50.500543 12958 tablet_service.cc:1467] Processing CreateTablet for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 (DEFAULT_TABLE table=default.Table_cd37c88af8ee458ea4d03e8901f98d9f [id=cc1abddccb024092b63578c266f7ff9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.501411 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4cdb7dd4e44b45963cf93eaefe23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.506350 14211 raft_consensus.cc:482] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.506507 13096 tablet_service.cc:1467] Processing CreateTablet for tablet e0fbbc77c0b64c2fa3d125e074eecd87 (DEFAULT_TABLE table=default.Table_807ff2b24da64f7396c8dc15bdc46c88 [id=38c0ff2e5c6d457884dbddc850f4c7d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.506500 14211 raft_consensus.cc:504] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.507243 14211 leader_election.cc:288] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:50.507625 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 110a046c2db14758b8d28616ae63c18c (DEFAULT_TABLE table=default.Table_50e723eab6834893b76afae99961fb99 [id=b37f4ffab2074941acbeea4fdc0c2f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.508000 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fbbc77c0b64c2fa3d125e074eecd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.508103 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110a046c2db14758b8d28616ae63c18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.509155 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 28194038348545dbbae94e20fed2e092 (DEFAULT_TABLE table=default.Table_95c37af6ef334405b443163c4eb60668 [id=2eebc48dfe464f9ba15aec649e8f9d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.509577 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28194038348545dbbae94e20fed2e092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.510392 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 91c73fd47e1d4205b697570b68e69870 (DEFAULT_TABLE table=default.Table_4607cd1fdeea4e3fa1d76a329c0b0191 [id=6a0f034b7a6c409cabb305e961ab987b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.510684 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c73fd47e1d4205b697570b68e69870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.513564 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 110a046c2db14758b8d28616ae63c18c (DEFAULT_TABLE table=default.Table_50e723eab6834893b76afae99961fb99 [id=b37f4ffab2074941acbeea4fdc0c2f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.513586 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 28194038348545dbbae94e20fed2e092 (DEFAULT_TABLE table=default.Table_95c37af6ef334405b443163c4eb60668 [id=2eebc48dfe464f9ba15aec649e8f9d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.513911 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110a046c2db14758b8d28616ae63c18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.513957 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28194038348545dbbae94e20fed2e092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.516286 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 110a046c2db14758b8d28616ae63c18c (DEFAULT_TABLE table=default.Table_50e723eab6834893b76afae99961fb99 [id=b37f4ffab2074941acbeea4fdc0c2f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.516635 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110a046c2db14758b8d28616ae63c18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.517742 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1615f6167aae4dc096b9f563ab5e6215" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:50.518328 13058 leader_election.cc:341] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.518582 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1615f6167aae4dc096b9f563ab5e6215" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:50.519126 13057 leader_election.cc:341] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.519219 13057 leader_election.cc:302] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.519742 14211 raft_consensus.cc:2726] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.522900 12811 tablet_service.cc:1467] Processing CreateTablet for tablet e39b7a5713914234b5b3858ef254394c (DEFAULT_TABLE table=default.Table_491b0792c74c4700b9c1b789c3ad16d9 [id=cb87e13b10794ab6a279c31169d8afb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.523358 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39b7a5713914234b5b3858ef254394c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.523433 13094 tablet_service.cc:1467] Processing CreateTablet for tablet e39b7a5713914234b5b3858ef254394c (DEFAULT_TABLE table=default.Table_491b0792c74c4700b9c1b789c3ad16d9 [id=cb87e13b10794ab6a279c31169d8afb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.523787 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39b7a5713914234b5b3858ef254394c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.523833 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 28194038348545dbbae94e20fed2e092 (DEFAULT_TABLE table=default.Table_95c37af6ef334405b443163c4eb60668 [id=2eebc48dfe464f9ba15aec649e8f9d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.524245 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28194038348545dbbae94e20fed2e092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.527846 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 91c73fd47e1d4205b697570b68e69870 (DEFAULT_TABLE table=default.Table_4607cd1fdeea4e3fa1d76a329c0b0191 [id=6a0f034b7a6c409cabb305e961ab987b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.528131 14211 raft_consensus.cc:482] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.528208 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c73fd47e1d4205b697570b68e69870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.528261 14211 raft_consensus.cc:504] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.528955 14211 leader_election.cc:288] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:50.531546 12962 tablet_service.cc:1467] Processing CreateTablet for tablet e39b7a5713914234b5b3858ef254394c (DEFAULT_TABLE table=default.Table_491b0792c74c4700b9c1b789c3ad16d9 [id=cb87e13b10794ab6a279c31169d8afb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.532852 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:50.533492 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39b7a5713914234b5b3858ef254394c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.534915 13243 ts_tablet_manager.cc:1888] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.535006 13243 log.cc:1196] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/034d6309de0740d0a5806b3847e334cb
I20240506 19:56:50.536424 13243 ts_tablet_manager.cc:1909] T 034d6309de0740d0a5806b3847e334cb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:50.536864 13058 leader_election.cc:341] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:50.538734 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 673dc7c83f8748a09c41d51b42214aaf consensus state has no leader
I20240506 19:56:50.540271 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]). Expected version 3 got 0
I20240506 19:56:50.541711 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 654) took 3467 ms. Trace:
I20240506 19:56:50.541818 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:50.541810 13242 rpcz_store.cc:276] 0506 19:56:47.073937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.073999 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:50.277889 (+3203890us) ts_tablet_manager.cc:1120] Deleting tablet a36b4d51b100486c831a41a52d81fd66
0506 19:56:50.281024 (+ 3135us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.380201 (+ 99177us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.499563 (+119362us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.541687 (+ 42124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41249,"tablet-delete.queue_time_us":3203616}
I20240506 19:56:50.542306 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a36b4d51b100486c831a41a52d81fd66 (table default.Table_e5d18039b7f04edaa0c985be5260be35 [id=7a6f6d0d394340ea93a13720ea7b769e]) successfully deleted
W20240506 19:56:50.543476 13057 leader_election.cc:341] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.543272 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 655) took 3398 ms. Trace:
I20240506 19:56:50.543596 13057 leader_election.cc:302] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.543599 13242 rpcz_store.cc:276] 0506 19:56:47.144726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.144788 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:50.543139 (+3398351us) ts_tablet_manager.cc:1120] Deleting tablet 01c32eba2031446ab419cac64068068e
0506 19:56:50.543150 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.543253 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3398134}
I20240506 19:56:50.543860 14211 raft_consensus.cc:2726] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.544173 13242 tablet_replica.cc:330] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:50.541162 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_951df9b3f85e416893c271dc056e57ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.544626 13242 raft_consensus.cc:2229] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.544771 13242 raft_consensus.cc:2258] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:50.544975 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_951df9b3f85e416893c271dc056e57ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.545337 13242 ts_tablet_manager.cc:1875] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.548399 12819 tablet_service.cc:1467] Processing CreateTablet for tablet c725cce0d2844d0a894bcc2a6aa4b3d3 (DEFAULT_TABLE table=default.Table_a7afaab4e68b4077a2876e0a0c12a73b [id=99c895fce27a4cfd97900cf3c10d7633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.548528 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 91c73fd47e1d4205b697570b68e69870 (DEFAULT_TABLE table=default.Table_4607cd1fdeea4e3fa1d76a329c0b0191 [id=6a0f034b7a6c409cabb305e961ab987b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.549116 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c73fd47e1d4205b697570b68e69870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.552019 13101 tablet_service.cc:1467] Processing CreateTablet for tablet c725cce0d2844d0a894bcc2a6aa4b3d3 (DEFAULT_TABLE table=default.Table_a7afaab4e68b4077a2876e0a0c12a73b [id=99c895fce27a4cfd97900cf3c10d7633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.552364 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725cce0d2844d0a894bcc2a6aa4b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.553169 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725cce0d2844d0a894bcc2a6aa4b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.557015 13242 ts_tablet_manager.cc:1888] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.557147 13242 log.cc:1196] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/07755e37a20646a3b1a7b900d54a077c
I20240506 19:56:50.557653 14080 raft_consensus.cc:482] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.557813 14080 raft_consensus.cc:504] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.558501 14080 leader_election.cc:288] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.559635 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f7171d06948388bc0fdb3514202ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:50.559837 13120 raft_consensus.cc:2454] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:50.560365 12922 leader_election.cc:302] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:50.560613 14080 raft_consensus.cc:2781] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:50.560681 14080 raft_consensus.cc:482] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:50.560737 14080 raft_consensus.cc:3037] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.560781 12813 tablet_service.cc:1467] Processing CreateTablet for tablet fd1f00e8620640b08cc054c043900738 (DEFAULT_TABLE table=default.Table_61659b4f08a1437f953f0e5345dcfc8f [id=7249e39100454579b86f755187e5aeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.561192 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1f00e8620640b08cc054c043900738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.562348 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 110a046c2db14758b8d28616ae63c18c
I20240506 19:56:50.563218 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0f8f0a413eab460aa84362ac194994e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.563748 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f8f0a413eab460aa84362ac194994e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.563766 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 110a046c2db14758b8d28616ae63c18c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.564013 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 110a046c2db14758b8d28616ae63c18c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.565004 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 269210fbc3d4446181b123f200aae472
I20240506 19:56:50.566279 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0bc56299ed854d6c8ef201127692fd60
I20240506 19:56:50.566486 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 269210fbc3d4446181b123f200aae472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.567209 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4c2eb5dac9d34fb09119868bf5a85803
I20240506 19:56:50.567296 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc56299ed854d6c8ef201127692fd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.568210 13091 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2eb5dac9d34fb09119868bf5a85803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.568324 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f7171d06948388bc0fdb3514202ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:50.568567 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc56299ed854d6c8ef201127692fd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.568800 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 269210fbc3d4446181b123f200aae472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:50.568885 12921 leader_election.cc:341] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.569069 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet bd26e6b8115644cabe6532c7be1bbe95 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:50.570431 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_da2ec44bc10845a7a4fbf93f4916c652" } new_table_name: "default.Table_da8cee75a0c844e1a4956550119cc252" modify_external_catalogs: true
I20240506 19:56:50.572088 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1e64cb2f31b64b07848c2dfed18ab9f1
I20240506 19:56:50.573371 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.573668 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.573863 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7ecd2c36503246d8992d73e00cf06129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.574366 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ecd2c36503246d8992d73e00cf06129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.576043 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_303d9190978b40a89152b877de75abb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.576531 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_303d9190978b40a89152b877de75abb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.577263 13103 tablet_service.cc:1467] Processing CreateTablet for tablet fd1f00e8620640b08cc054c043900738 (DEFAULT_TABLE table=default.Table_61659b4f08a1437f953f0e5345dcfc8f [id=7249e39100454579b86f755187e5aeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.577706 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1f00e8620640b08cc054c043900738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.578471 13244 tablet_bootstrap.cc:492] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.578673 13244 ts_tablet_manager.cc:1367] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.210s user 0.018s sys 0.018s
I20240506 19:56:50.579253 12963 tablet_service.cc:1467] Processing CreateTablet for tablet c725cce0d2844d0a894bcc2a6aa4b3d3 (DEFAULT_TABLE table=default.Table_a7afaab4e68b4077a2876e0a0c12a73b [id=99c895fce27a4cfd97900cf3c10d7633]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.579386 13244 raft_consensus.cc:348] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.579555 13244 raft_consensus.cc:374] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.579607 13244 raft_consensus.cc:729] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.579658 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725cce0d2844d0a894bcc2a6aa4b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.579880 13244 consensus_queue.cc:260] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.580366 13244 ts_tablet_manager.cc:1398] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:50.581720 13244 tablet_bootstrap.cc:492] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.583554 13244 tablet_bootstrap.cc:654] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.583842 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 269210fbc3d4446181b123f200aae472 (DEFAULT_TABLE table=default.Table_94fff765fd9146e5a2920c3702f5c8af [id=6c09bd4bc1514b43b9dd19458d6b73a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.589931 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5893f3f184b247949785e248aa0e53c9" } new_table_name: "default.Table_110baf1a2d08406abaddd6e76dd187a4" modify_external_catalogs: true
I20240506 19:56:50.590379 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 3b74c481671149a6bc6b138b8b2d7e42 (DEFAULT_TABLE table=default.Table_5b7eae44ba9746f9b088affbf1a88cdb [id=273f7fc51349495f90765ff0b2ba2c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.591499 13244 tablet_bootstrap.cc:492] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.591620 13244 ts_tablet_manager.cc:1367] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240506 19:56:50.592255 13244 raft_consensus.cc:348] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.592434 13244 raft_consensus.cc:374] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.592478 13244 raft_consensus.cc:729] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.592670 13244 consensus_queue.cc:260] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.593151 13244 ts_tablet_manager.cc:1398] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 19:56:50.594069 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2d5a4569d401426eb881b08e83c66e86" } new_table_name: "default.Table_de7835c60b14419892d8827447c27f2e" modify_external_catalogs: true
I20240506 19:56:50.595134 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 3b74c481671149a6bc6b138b8b2d7e42 (DEFAULT_TABLE table=default.Table_5b7eae44ba9746f9b088affbf1a88cdb [id=273f7fc51349495f90765ff0b2ba2c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.595726 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b74c481671149a6bc6b138b8b2d7e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.601105 13242 ts_tablet_manager.cc:1909] T 07755e37a20646a3b1a7b900d54a077c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:50.597502 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2eb5dac9d34fb09119868bf5a85803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.602352 12965 tablet_service.cc:1467] Processing CreateTablet for tablet fd1f00e8620640b08cc054c043900738 (DEFAULT_TABLE table=default.Table_61659b4f08a1437f953f0e5345dcfc8f [id=7249e39100454579b86f755187e5aeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.602891 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1f00e8620640b08cc054c043900738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.605754 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 110a046c2db14758b8d28616ae63c18c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.605870 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 269210fbc3d4446181b123f200aae472 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.606264 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc56299ed854d6c8ef201127692fd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.606583 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2eb5dac9d34fb09119868bf5a85803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.606873 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.607369 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 269210fbc3d4446181b123f200aae472 (DEFAULT_TABLE table=default.Table_94fff765fd9146e5a2920c3702f5c8af [id=6c09bd4bc1514b43b9dd19458d6b73a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.607858 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269210fbc3d4446181b123f200aae472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.610726 13096 tablet_service.cc:1467] Processing CreateTablet for tablet dcdb37b1d46f44f48b3c473956c1937e (DEFAULT_TABLE table=default.Table_71a4779621224f0aa1c1b137ad0de871 [id=fd183a93acb14d58b828ecc56e2002d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.611169 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdb37b1d46f44f48b3c473956c1937e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.611837 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 3b74c481671149a6bc6b138b8b2d7e42 (DEFAULT_TABLE table=default.Table_5b7eae44ba9746f9b088affbf1a88cdb [id=273f7fc51349495f90765ff0b2ba2c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.611910 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_86e7dc4a07f542679409dcc2a3187f46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.612174 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b74c481671149a6bc6b138b8b2d7e42. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.612510 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86e7dc4a07f542679409dcc2a3187f46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.616645 12827 tablet_service.cc:1467] Processing CreateTablet for tablet cf97a98927784b7080b8f30b33fe91bf (DEFAULT_TABLE table=default.Table_803e3127dd644ef7a78b9e24616a7d66 [id=a6da1ed0f23d4a1482344a047b52f642]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:50.617035 13058 consensus_peers.cc:485] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.619761 12813 tablet_service.cc:1467] Processing CreateTablet for tablet dcdb37b1d46f44f48b3c473956c1937e (DEFAULT_TABLE table=default.Table_71a4779621224f0aa1c1b137ad0de871 [id=fd183a93acb14d58b828ecc56e2002d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.620123 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 269210fbc3d4446181b123f200aae472 (DEFAULT_TABLE table=default.Table_94fff765fd9146e5a2920c3702f5c8af [id=6c09bd4bc1514b43b9dd19458d6b73a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.620632 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269210fbc3d4446181b123f200aae472. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.622059 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11b00381c6aa42c9b969410b51ebe81e consensus state has no leader
I20240506 19:56:50.622843 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:56:50.623374 12516 catalog_manager.cc:6554] ReplaceTablet: tablet bd26e6b8115644cabe6532c7be1bbe95 deleted and replaced by tablet a152a0f6a9ed4aa5b74fe5a53688331d
I20240506 19:56:50.624235 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 with delete_type TABLET_DATA_DELETED (replaced by tablet a152a0f6a9ed4aa5b74fe5a53688331d) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.627022 12967 tablet_service.cc:1467] Processing CreateTablet for tablet cf97a98927784b7080b8f30b33fe91bf (DEFAULT_TABLE table=default.Table_803e3127dd644ef7a78b9e24616a7d66 [id=a6da1ed0f23d4a1482344a047b52f642]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.627424 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf97a98927784b7080b8f30b33fe91bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.627368 12948 tablet_service.cc:1467] Processing CreateTablet for tablet dcdb37b1d46f44f48b3c473956c1937e (DEFAULT_TABLE table=default.Table_71a4779621224f0aa1c1b137ad0de871 [id=fd183a93acb14d58b828ecc56e2002d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.627663 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdb37b1d46f44f48b3c473956c1937e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.631083 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 with delete_type TABLET_DATA_DELETED (replaced by tablet a152a0f6a9ed4aa5b74fe5a53688331d) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.631520 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8c2dd5f0bc85416b9c98a687c245f5b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.632192 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c2dd5f0bc85416b9c98a687c245f5b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.637133 13094 tablet_service.cc:1467] Processing CreateTablet for tablet cf97a98927784b7080b8f30b33fe91bf (DEFAULT_TABLE table=default.Table_803e3127dd644ef7a78b9e24616a7d66 [id=a6da1ed0f23d4a1482344a047b52f642]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.637430 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 14cffcd4198846ddadae19cd5e80b318 (DEFAULT_TABLE table=default.Table_a010c4cc7a5348d79e8c910419d72c98 [id=1ee0a9b9312f4deda04f6199c1014af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.637615 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf97a98927784b7080b8f30b33fe91bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.639537 14080 raft_consensus.cc:504] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.640208 14080 leader_election.cc:288] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.640964 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f7171d06948388bc0fdb3514202ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:50.641531 13243 tablet_replica.cc:330] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.641703 13243 raft_consensus.cc:2229] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.641840 13243 raft_consensus.cc:2258] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.642223 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f7171d06948388bc0fdb3514202ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:50.642393 13120 raft_consensus.cc:3037] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.642481 13243 ts_tablet_manager.cc:1875] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.642594 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 14cffcd4198846ddadae19cd5e80b318 (DEFAULT_TABLE table=default.Table_a010c4cc7a5348d79e8c910419d72c98 [id=1ee0a9b9312f4deda04f6199c1014af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.642937 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cffcd4198846ddadae19cd5e80b318. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.641800 12921 leader_election.cc:341] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:50.645663 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63decce057244d72b4a812368594fcfb has no consensus state
I20240506 19:56:50.645728 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 35198d5786c14de89c9751a0372ba91d (DEFAULT_TABLE table=default.Table_52654f2617b946ad99d761a8a1b45c4d [id=82b3a328a4e4447ba247fa8cf6b179c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.645987 13120 raft_consensus.cc:2454] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:50.646260 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c93117bdcf44c719c2e3588de538e96 consensus state has no leader
I20240506 19:56:50.646462 12922 leader_election.cc:302] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.646903 14080 raft_consensus.cc:2781] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:50.647022 14080 raft_consensus.cc:686] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:50.647253 14080 consensus_queue.cc:237] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:50.649212 13058 consensus_peers.cc:485] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.649878 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 854cb84e650348d7b8a633c977d49c53 (DEFAULT_TABLE table=default.Table_86885529e3ce46e792201516f327c805 [id=7b921f73ecf9462691e34bda425719c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.653942 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 14cffcd4198846ddadae19cd5e80b318 (DEFAULT_TABLE table=default.Table_a010c4cc7a5348d79e8c910419d72c98 [id=1ee0a9b9312f4deda04f6199c1014af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.654438 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cffcd4198846ddadae19cd5e80b318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.655793 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 034d6309de0740d0a5806b3847e334cb (table default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]) successfully deleted
I20240506 19:56:50.656212 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 35198d5786c14de89c9751a0372ba91d (DEFAULT_TABLE table=default.Table_52654f2617b946ad99d761a8a1b45c4d [id=82b3a328a4e4447ba247fa8cf6b179c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.657492 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35198d5786c14de89c9751a0372ba91d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.659044 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f3bbf5a4326448d684fcb3f138dcf96d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bbf5a4326448d684fcb3f138dcf96d already in progress: creating tablet
I20240506 19:56:50.659763 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 09a396f5c8684be7967be14b1b15195b (DEFAULT_TABLE table=default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.658829 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 854cb84e650348d7b8a633c977d49c53 (DEFAULT_TABLE table=default.Table_86885529e3ce46e792201516f327c805 [id=7b921f73ecf9462691e34bda425719c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.662839 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 09a396f5c8684be7967be14b1b15195b (DEFAULT_TABLE table=default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.663348 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854cb84e650348d7b8a633c977d49c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.663548 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a396f5c8684be7967be14b1b15195b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.664309 12961 tablet_service.cc:1467] Processing CreateTablet for tablet e0fbbc77c0b64c2fa3d125e074eecd87 (DEFAULT_TABLE table=default.Table_807ff2b24da64f7396c8dc15bdc46c88 [id=38c0ff2e5c6d457884dbddc850f4c7d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.664673 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fbbc77c0b64c2fa3d125e074eecd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.666003 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 35198d5786c14de89c9751a0372ba91d (DEFAULT_TABLE table=default.Table_52654f2617b946ad99d761a8a1b45c4d [id=82b3a328a4e4447ba247fa8cf6b179c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.666014 12828 tablet_service.cc:1467] Processing CreateTablet for tablet d842127c078f4d02bca0c63531e72c0d (DEFAULT_TABLE table=default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b [id=66878162ce704b9da1844171993b6f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.666349 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35198d5786c14de89c9751a0372ba91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.664674 13243 ts_tablet_manager.cc:1888] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.667057 13243 log.cc:1196] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/bafd3c4706f74e7da36a8f3538760f43
I20240506 19:56:50.667361 13243 ts_tablet_manager.cc:1909] T bafd3c4706f74e7da36a8f3538760f43 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.667773 13243 tablet_replica.cc:330] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.667963 13243 raft_consensus.cc:2229] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
W20240506 19:56:50.668058 13058 consensus_peers.cc:485] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.668315 13243 raft_consensus.cc:2258] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.668808 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 854cb84e650348d7b8a633c977d49c53 (DEFAULT_TABLE table=default.Table_86885529e3ce46e792201516f327c805 [id=7b921f73ecf9462691e34bda425719c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.669131 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854cb84e650348d7b8a633c977d49c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.672367 13243 ts_tablet_manager.cc:1875] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.672401 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 01da81cb9ee840ed9e6640af305eaecc (DEFAULT_TABLE table=default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.675201 13091 tablet_service.cc:1467] Processing CreateTablet for tablet d842127c078f4d02bca0c63531e72c0d (DEFAULT_TABLE table=default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b [id=66878162ce704b9da1844171993b6f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.675730 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d842127c078f4d02bca0c63531e72c0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:50.677361 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46fd9906504e4b95a5b05b01dc3ea1c0 already in progress: creating tablet
I20240506 19:56:50.681849 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 01da81cb9ee840ed9e6640af305eaecc (DEFAULT_TABLE table=default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.682261 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01da81cb9ee840ed9e6640af305eaecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.657934 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e74f7640674041f292cd9a7b4ffb3e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.685627 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 with delete_type TABLET_DATA_DELETED (replaced by tablet a152a0f6a9ed4aa5b74fe5a53688331d) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:50.686080 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e74f7640674041f292cd9a7b4ffb3e9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.687877 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4213fa718b491881f304b4e8f0bf7a (DEFAULT_TABLE table=default.Table_fb883920a7c94fd4a786a7b48237c418 [id=f29370997a8747a3b90e53462f251cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.688239 14223 raft_consensus.cc:482] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.688376 14223 raft_consensus.cc:504] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.689023 14223 leader_election.cc:288] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:50.689889 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acde41b2c03440a89243aae008779e1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:50.690081 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8321457717fd4ad28ae1292d9e9a8669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8321457717fd4ad28ae1292d9e9a8669 already in progress: creating tablet
W20240506 19:56:50.690495 13058 leader_election.cc:341] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.692199 13245 tablet_bootstrap.cc:492] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.692391 13245 ts_tablet_manager.cc:1367] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.364s user 0.023s sys 0.004s
I20240506 19:56:50.693158 13245 raft_consensus.cc:348] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.693416 13245 raft_consensus.cc:374] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.693524 13245 raft_consensus.cc:729] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.693826 13245 consensus_queue.cc:260] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.695034 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet bafd3c4706f74e7da36a8f3538760f43 (table default.Table_162d0c1d88d04a11b0015afaea52e19f [id=5a2afe178f674b29834ccf87ec1ed218]) successfully deleted
I20240506 19:56:50.704777 13245 ts_tablet_manager.cc:1398] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240506 19:56:50.705811 13246 ts_tablet_manager.cc:1888] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.705916 13246 log.cc:1196] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ac9ecd2e034e49e49772db7435da3c81
I20240506 19:56:50.706200 13246 ts_tablet_manager.cc:1909] T ac9ecd2e034e49e49772db7435da3c81 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:50.709415 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.712407 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 746) took 2473 ms. Trace:
I20240506 19:56:50.712517 13246 rpcz_store.cc:276] 0506 19:56:48.238497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.250005 (+ 11508us) service_pool.cc:225] Handling call
0506 19:56:50.280534 (+2030529us) ts_tablet_manager.cc:1120] Deleting tablet ac9ecd2e034e49e49772db7435da3c81
0506 19:56:50.367088 (+ 86554us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.371150 (+ 4062us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.705708 (+334558us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.712384 (+ 6676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029322}
I20240506 19:56:50.713156 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ac9ecd2e034e49e49772db7435da3c81 (table default.Table_9608d1af549c42e28d02df6bd141adf8 [id=feed002cdb9d4296a14ca76514e18ed4]) successfully deleted
I20240506 19:56:50.714562 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 09a396f5c8684be7967be14b1b15195b (DEFAULT_TABLE table=default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.715049 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a396f5c8684be7967be14b1b15195b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.718139 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4213fa718b491881f304b4e8f0bf7a (DEFAULT_TABLE table=default.Table_fb883920a7c94fd4a786a7b48237c418 [id=f29370997a8747a3b90e53462f251cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.722309 13245 tablet_bootstrap.cc:492] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:50.723017 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acde41b2c03440a89243aae008779e1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:50.723523 13057 leader_election.cc:341] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.723641 13057 leader_election.cc:302] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.723806 13245 tablet_bootstrap.cc:654] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.724071 14211 raft_consensus.cc:2726] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.732193 12952 tablet_service.cc:1467] Processing CreateTablet for tablet d842127c078f4d02bca0c63531e72c0d (DEFAULT_TABLE table=default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b [id=66878162ce704b9da1844171993b6f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.733448 13245 tablet_bootstrap.cc:492] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.733628 13245 ts_tablet_manager.cc:1367] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20240506 19:56:50.734561 13245 raft_consensus.cc:348] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.734813 13245 raft_consensus.cc:374] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.734885 13245 raft_consensus.cc:729] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.735370 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4213fa718b491881f304b4e8f0bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.735358 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 01da81cb9ee840ed9e6640af305eaecc (DEFAULT_TABLE table=default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.736231 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d842127c078f4d02bca0c63531e72c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.740528 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_61659b4f08a1437f953f0e5345dcfc8f" } new_table_name: "default.Table_a0ca5a4363dd40c6840fe7989c885cb0" modify_external_catalogs: true
I20240506 19:56:50.741123 13249 tablet_bootstrap.cc:492] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:50.741261 13249 ts_tablet_manager.cc:1367] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.269s user 0.017s sys 0.011s
I20240506 19:56:50.741386 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 747) took 2411 ms. Trace:
I20240506 19:56:50.741596 13246 rpcz_store.cc:276] 0506 19:56:48.329986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.330053 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:50.712803 (+2382750us) ts_tablet_manager.cc:1120] Deleting tablet bd3130bc41084bfaabaaecc0985e378d
0506 19:56:50.735056 (+ 22253us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.741364 (+ 6308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2382557}
I20240506 19:56:50.742010 13246 tablet_replica.cc:330] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:50.741968 13249 raft_consensus.cc:348] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.742138 13249 raft_consensus.cc:374] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.742194 13249 raft_consensus.cc:729] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.742203 13246 raft_consensus.cc:2229] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.742350 13246 raft_consensus.cc:2258] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:50.742321 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bd3130bc41084bfaabaaecc0985e378d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd3130bc41084bfaabaaecc0985e378d already in progress: creating tablet
I20240506 19:56:50.742408 13249 consensus_queue.cc:260] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.742965 13249 ts_tablet_manager.cc:1398] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:50.743036 13246 ts_tablet_manager.cc:1875] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.748476 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 996eb97639464e0bb976f2b1ecbbb73b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:50.749732 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269210fbc3d4446181b123f200aae472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.753537 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b74c481671149a6bc6b138b8b2d7e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.755502 13244 tablet_bootstrap.cc:492] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.757277 13244 tablet_bootstrap.cc:654] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.757603 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf97a98927784b7080b8f30b33fe91bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.761438 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdb37b1d46f44f48b3c473956c1937e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.765415 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cffcd4198846ddadae19cd5e80b318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.772729 14069 raft_consensus.cc:482] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.772888 14069 raft_consensus.cc:504] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.773553 14069 leader_election.cc:288] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.773700 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_491916e47bc640f199cb29f14c922469" } modify_external_catalogs: true
I20240506 19:56:50.773962 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_491916e47bc640f199cb29f14c922469" } modify_external_catalogs: true
I20240506 19:56:50.774394 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796628a0859c4511a8cb35e495dc8886" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:50.779038 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b3ed0b56aa478abb44347afca8fcf6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9296bb1104b4cbe279eecd3683796) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.778900 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c649504434cb4d69ac0d74733ff3591e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.779913 13245 consensus_queue.cc:260] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:50.780263 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c649504434cb4d69ac0d74733ff3591e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.780467 13245 ts_tablet_manager.cc:1398] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20240506 19:56:50.782240 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_47913566bb1846d19fddafa8a3a65bd6" } modify_external_catalogs: true
I20240506 19:56:50.782469 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_47913566bb1846d19fddafa8a3a65bd6" } modify_external_catalogs: true
I20240506 19:56:50.784106 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_95be5bd2c51b4613b0152034aae6350d" } modify_external_catalogs: true
I20240506 19:56:50.784348 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_95be5bd2c51b4613b0152034aae6350d" } modify_external_catalogs: true
I20240506 19:56:50.785308 14080 raft_consensus.cc:482] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.785459 14080 raft_consensus.cc:504] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.786195 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_74493d529d484630900cd65b7b1777da" } modify_external_catalogs: true
I20240506 19:56:50.786401 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_74493d529d484630900cd65b7b1777da" } modify_external_catalogs: true
I20240506 19:56:50.786466 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f56d2180b1472a8168fb3a63589065" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:50.787017 12921 leader_election.cc:341] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.787875 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f56d2180b1472a8168fb3a63589065" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:50.788107 13120 raft_consensus.cc:2454] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:50.788053 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a410467878c2481f86118d12ccd06f28" } modify_external_catalogs: true
I20240506 19:56:50.788318 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a410467878c2481f86118d12ccd06f28" } modify_external_catalogs: true
I20240506 19:56:50.788517 12922 leader_election.cc:302] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.788797 14250 raft_consensus.cc:2781] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:50.788873 14250 raft_consensus.cc:482] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:50.788936 14250 raft_consensus.cc:3037] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.790066 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4450908020fd475c8c535415af0a8925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.790977 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4450908020fd475c8c535415af0a8925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.791604 14250 raft_consensus.cc:504] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.792210 14250 leader_election.cc:288] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.793391 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f56d2180b1472a8168fb3a63589065" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:50.793936 12921 leader_election.cc:341] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.795248 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01da81cb9ee840ed9e6640af305eaecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.796260 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f56d2180b1472a8168fb3a63589065" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:50.796463 13120 raft_consensus.cc:3037] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.796555 13249 tablet_bootstrap.cc:492] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:50.799141 13120 raft_consensus.cc:2454] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:50.799794 12922 leader_election.cc:302] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:50.799918 13057 consensus_peers.cc:485] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 607b3fc2c17b4793af9b84c19066ec5e. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:50.800549 14249 raft_consensus.cc:2781] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:50.800657 14249 raft_consensus.cc:686] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:50.800891 14249 consensus_queue.cc:237] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:50.801625 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd1f00e8620640b08cc054c043900738 consensus state has no leader
I20240506 19:56:50.803611 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35198d5786c14de89c9751a0372ba91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.804639 14069 raft_consensus.cc:482] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.804770 14069 raft_consensus.cc:504] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.805389 14069 leader_election.cc:288] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.805886 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_8c81322d630a455595f418e3a97b441e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.806099 12781 leader_election.cc:341] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 796628a0859c4511a8cb35e495dc8886
I20240506 19:56:50.806200 12781 leader_election.cc:302] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:50.806572 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c81322d630a455595f418e3a97b441e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:50.806744 12781 leader_election.cc:341] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 1372be71e1d141ed89e58cdb806431a4
I20240506 19:56:50.806906 14069 raft_consensus.cc:2726] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.809577 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854cb84e650348d7b8a633c977d49c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.810688 13249 tablet_bootstrap.cc:654] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:50.815569 12783 leader_election.cc:341] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1372be71e1d141ed89e58cdb806431a4
I20240506 19:56:50.815703 12783 leader_election.cc:302] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.816061 14069 raft_consensus.cc:2726] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.816604 13249 tablet_bootstrap.cc:492] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:50.816830 13249 ts_tablet_manager.cc:1367] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240506 19:56:50.818006 13249 raft_consensus.cc:348] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.820209 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f0a5ff6d430282256e9bddc186b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 99f4861a01404663870ce91978c8af4f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.822342 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a03f574aec984b83ab596e92a6f323fc
I20240506 19:56:50.822784 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a396f5c8684be7967be14b1b15195b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.823405 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_de4daf2bb6824e22ab5ee4ae55a493d7" } new_table_name: "default.Table_8c32c6726a7f4aa2867ec4f2594083d5" modify_external_catalogs: true
I20240506 19:56:50.825559 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d842127c078f4d02bca0c63531e72c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.825407 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2a11d75c7230425387b56ecf4c87a109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.825930 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a11d75c7230425387b56ecf4c87a109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.827616 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b3e50bde3564493eb7267acb42cb8592
I20240506 19:56:50.828224 13245 tablet_bootstrap.cc:492] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:50.828289 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7f8989f02f25418ea9f824b3e5eeddd6
I20240506 19:56:50.828389 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01da81cb9ee840ed9e6640af305eaecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.828963 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9bb90bf767c94d73ace6411cfd31d65c
I20240506 19:56:50.829566 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5b15189e5cb54684a0f62308c2e5f131
I20240506 19:56:50.830701 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 996eb97639464e0bb976f2b1ecbbb73b
I20240506 19:56:50.831106 12518 catalog_manager.cc:6554] ReplaceTablet: tablet 996eb97639464e0bb976f2b1ecbbb73b deleted and replaced by tablet c727c9c20df34c4aa5c44b70add0d6b6
I20240506 19:56:50.833516 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4213fa718b491881f304b4e8f0bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.837985 13245 tablet_bootstrap.cc:654] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.841992 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8989f02f25418ea9f824b3e5eeddd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.841992 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.842479 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet a03f574aec984b83ab596e92a6f323fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.843323 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 996eb97639464e0bb976f2b1ecbbb73b with delete_type TABLET_DATA_DELETED (replaced by tablet c727c9c20df34c4aa5c44b70add0d6b6) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.844364 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_84cf0c067fa84ec2b9b35f2f981d4671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.845055 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84cf0c067fa84ec2b9b35f2f981d4671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.846508 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9caff027d79b4a8cbdde4ba6074c1e90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.847153 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9caff027d79b4a8cbdde4ba6074c1e90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.847913 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 656) took 3685 ms. Trace:
I20240506 19:56:50.848011 13242 rpcz_store.cc:276] 0506 19:56:47.162113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.162173 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:50.544129 (+3381956us) ts_tablet_manager.cc:1120] Deleting tablet 07755e37a20646a3b1a7b900d54a077c
0506 19:56:50.544137 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.554133 (+ 9996us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.556923 (+ 2790us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.847881 (+290958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280,"tablet-delete.queue_time_us":3381757}
I20240506 19:56:50.847158 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 996eb97639464e0bb976f2b1ecbbb73b with delete_type TABLET_DATA_DELETED (replaced by tablet c727c9c20df34c4aa5c44b70add0d6b6) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.848388 13242 tablet_replica.cc:330] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:50.848817 13242 raft_consensus.cc:2229] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.849005 13242 raft_consensus.cc:2258] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.848958 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 07755e37a20646a3b1a7b900d54a077c (table default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]) successfully deleted
I20240506 19:56:50.849632 13242 ts_tablet_manager.cc:1875] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.850559 14069 raft_consensus.cc:482] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.850674 14069 raft_consensus.cc:504] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.851286 14069 leader_election.cc:288] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:50.852268 12781 leader_election.cc:341] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d6e54543d1544b1c8cff3e890e787337
W20240506 19:56:50.852651 13057 consensus_peers.cc:485] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 034d6309de0740d0a5806b3847e334cb. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:50.853515 12783 leader_election.cc:341] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d6e54543d1544b1c8cff3e890e787337
I20240506 19:56:50.853610 12783 leader_election.cc:302] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:50.853825 14069 raft_consensus.cc:2726] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.855476 13245 tablet_bootstrap.cc:492] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.856586 13245 ts_tablet_manager.cc:1367] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240506 19:56:50.856803 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b15189e5cb54684a0f62308c2e5f131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.857043 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e50bde3564493eb7267acb42cb8592 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.858253 13242 ts_tablet_manager.cc:1888] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.858428 13242 log.cc:1196] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ea25fb8e42234b82885b8fb2275d4925
I20240506 19:56:50.858836 13242 ts_tablet_manager.cc:1909] T ea25fb8e42234b82885b8fb2275d4925 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:50.859225 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 657) took 3686 ms. Trace:
I20240506 19:56:50.859370 13242 rpcz_store.cc:276] 0506 19:56:47.172726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.172778 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:50.848341 (+3675563us) ts_tablet_manager.cc:1120] Deleting tablet ea25fb8e42234b82885b8fb2275d4925
0506 19:56:50.848349 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.852615 (+ 4266us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.858156 (+ 5541us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.859206 (+ 1050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3675357}
I20240506 19:56:50.859967 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ea25fb8e42234b82885b8fb2275d4925 (table default.Table_685d0e00c37640ab99bfe74c85c1c0f3 [id=8b3a851d7ee4438daf7440e2af8f9885]) successfully deleted
I20240506 19:56:50.860446 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 686) took 3540 ms. Trace:
I20240506 19:56:50.860654 13242 rpcz_store.cc:276] 0506 19:56:47.319779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.932637 (+612858us) service_pool.cc:225] Handling call
0506 19:56:50.860321 (+2927684us) ts_tablet_manager.cc:1120] Deleting tablet 1d53f3daaa2949a2b6883435f031dc37
0506 19:56:50.860332 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.860431 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927094}
I20240506 19:56:50.861402 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e50bde3564493eb7267acb42cb8592 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.862792 13246 ts_tablet_manager.cc:1888] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.862918 13246 log.cc:1196] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/378759ca310f453fba45d48879e2dd1a
I20240506 19:56:50.863230 13246 ts_tablet_manager.cc:1909] T 378759ca310f453fba45d48879e2dd1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:50.863517 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b15189e5cb54684a0f62308c2e5f131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:50.863755 13245 raft_consensus.cc:348] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.863950 13245 raft_consensus.cc:374] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.864014 13245 raft_consensus.cc:729] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.864123 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_78690cea0de8499bb46d757158d49954"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.864284 13245 consensus_queue.cc:260] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.864809 13245 ts_tablet_manager.cc:1398] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20240506 19:56:50.864993 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78690cea0de8499bb46d757158d49954 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.865092 13245 tablet_bootstrap.cc:492] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:50.866531 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8b082b3b3998450aac6c69f4dfd84585" } new_table_name: "default.Table_6a081bfc6c694bd2bb37cd06213e221a" modify_external_catalogs: true
I20240506 19:56:50.867863 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_260369fbb74b4623ae5419fdb3370d2a" } new_table_name: "default.Table_6de4fcdd30e14b65ac04120e9e628710" modify_external_catalogs: true
W20240506 19:56:50.871783 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1d53f3daaa2949a2b6883435f031dc37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d53f3daaa2949a2b6883435f031dc37 already in progress: creating tablet
I20240506 19:56:50.860989 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.861095 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8989f02f25418ea9f824b3e5eeddd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.861215 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet a03f574aec984b83ab596e92a6f323fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:50.872901 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 688) took 3541 ms. Trace:
I20240506 19:56:50.875313 13245 tablet_bootstrap.cc:654] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.863529 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 748) took 2525 ms. Trace:
I20240506 19:56:50.876348 13246 rpcz_store.cc:276] 0506 19:56:48.338482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.338559 (+ 77us) service_pool.cc:225] Handling call
0506 19:56:50.741964 (+2403405us) ts_tablet_manager.cc:1120] Deleting tablet 378759ca310f453fba45d48879e2dd1a
0506 19:56:50.741973 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.832377 (+ 90404us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.862683 (+ 30306us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.863515 (+ 832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2403202}
I20240506 19:56:50.876418 13242 rpcz_store.cc:276] 0506 19:56:47.331599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.933609 (+602010us) service_pool.cc:225] Handling call
0506 19:56:50.872755 (+2939146us) ts_tablet_manager.cc:1120] Deleting tablet 60d1f01c0edb4f6b8a621c1643ca1afb
0506 19:56:50.872768 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.872877 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2938878}
I20240506 19:56:50.877009 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 749) took 2507 ms. Trace:
I20240506 19:56:50.877099 13246 rpcz_store.cc:276] 0506 19:56:48.369368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.369871 (+ 503us) service_pool.cc:225] Handling call
0506 19:56:50.876855 (+2506984us) ts_tablet_manager.cc:1120] Deleting tablet 574dbbe8a9a94cd7944b652a68b77403
0506 19:56:50.876865 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.876988 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506763}
W20240506 19:56:50.877471 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 60d1f01c0edb4f6b8a621c1643ca1afb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d1f01c0edb4f6b8a621c1643ca1afb already in progress: creating tablet
W20240506 19:56:50.877995 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 574dbbe8a9a94cd7944b652a68b77403 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 574dbbe8a9a94cd7944b652a68b77403 already in progress: creating tablet
I20240506 19:56:50.878368 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 689) took 3546 ms. Trace:
I20240506 19:56:50.878423 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 750) took 2508 ms. Trace:
I20240506 19:56:50.878494 13246 rpcz_store.cc:276] 0506 19:56:48.369544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.370134 (+ 590us) service_pool.cc:225] Handling call
0506 19:56:50.878294 (+2508160us) ts_tablet_manager.cc:1120] Deleting tablet 7763959136c44674888916321bd6da24
0506 19:56:50.878307 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.878408 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2508016}
W20240506 19:56:50.878968 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7763959136c44674888916321bd6da24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7763959136c44674888916321bd6da24 already in progress: creating tablet
I20240506 19:56:50.878626 13242 rpcz_store.cc:276] 0506 19:56:47.331697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.933893 (+602196us) service_pool.cc:225] Handling call
0506 19:56:50.877947 (+2944054us) ts_tablet_manager.cc:1120] Deleting tablet 1b89fb24d94042e1bfa772265a991680
0506 19:56:50.877960 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.878070 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2943382}
I20240506 19:56:50.879956 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 690) took 3333 ms. Trace:
I20240506 19:56:50.880199 13246 tablet_replica.cc:330] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:50.880438 13246 raft_consensus.cc:2229] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.880631 13242 rpcz_store.cc:276] 0506 19:56:47.546199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.934603 (+388404us) service_pool.cc:225] Handling call
0506 19:56:50.879846 (+2945243us) ts_tablet_manager.cc:1120] Deleting tablet f759b560e2ab41a6a076976ccca811d1
0506 19:56:50.879856 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.879938 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2944427}
W20240506 19:56:50.883126 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f759b560e2ab41a6a076976ccca811d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f759b560e2ab41a6a076976ccca811d1 already in progress: creating tablet
I20240506 19:56:50.883554 13242 tablet_replica.cc:330] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:50.883957 13242 raft_consensus.cc:2229] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.884680 13242 raft_consensus.cc:2258] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.885504 13242 ts_tablet_manager.cc:1875] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.891444 13245 tablet_bootstrap.cc:492] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.891690 13245 ts_tablet_manager.cc:1367] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240506 19:56:50.892549 13245 raft_consensus.cc:348] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.892751 13245 raft_consensus.cc:374] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.892817 13245 raft_consensus.cc:729] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.893102 13245 consensus_queue.cc:260] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.893790 13245 ts_tablet_manager.cc:1398] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:50.894191 13245 tablet_bootstrap.cc:492] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:50.894220 14271 tablet.cc:1722] T 4e2c9f4baf55412d9bc00818a00c652d P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:50.902226 12848 raft_consensus.cc:3037] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.902621 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bd8cdb92bf4e49ffb08353c5af21f72e" } new_table_name: "default.Table_b35bb26efd3c4ca08971e6c1185a6c0b" modify_external_catalogs: true
I20240506 19:56:50.820781 14080 leader_election.cc:288] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.903880 13244 tablet_bootstrap.cc:492] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:50.904021 13244 ts_tablet_manager.cc:1367] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.030s
I20240506 19:56:50.904740 13249 raft_consensus.cc:374] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.904837 13249 raft_consensus.cc:729] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.905406 14280 tablet.cc:1722] T 4e2c9f4baf55412d9bc00818a00c652d P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:50.865409 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e50bde3564493eb7267acb42cb8592 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:50.865693 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 62601.5us
I20240506 19:56:50.865778 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8989f02f25418ea9f824b3e5eeddd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.865975 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b15189e5cb54684a0f62308c2e5f131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.866230 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.869066 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 996eb97639464e0bb976f2b1ecbbb73b with delete_type TABLET_DATA_DELETED (replaced by tablet c727c9c20df34c4aa5c44b70add0d6b6) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.863284 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4213fa718b491881f304b4e8f0bf7a (DEFAULT_TABLE table=default.Table_fb883920a7c94fd4a786a7b48237c418 [id=f29370997a8747a3b90e53462f251cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:50.908353 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4213fa718b491881f304b4e8f0bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:50.893726 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet a03f574aec984b83ab596e92a6f323fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.908881 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:50.906116 13249 consensus_queue.cc:260] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:50.912822 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fe8d4891c484627ac729971ff7aad3e consensus state has no leader
I20240506 19:56:50.914958 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_10658c410e1648df879bdf21d2293b2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.915639 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10658c410e1648df879bdf21d2293b2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.914469 13249 ts_tablet_manager.cc:1398] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.098s user 0.002s sys 0.000s
W20240506 19:56:50.917552 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e32a40abc4e74586af79474e4b1675c5 consensus state has no leader
I20240506 19:56:50.917987 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2a2d910a263840e7934d2972c9cd8a87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:50.918573 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a2d910a263840e7934d2972c9cd8a87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.895628 13245 tablet_bootstrap.cc:654] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:50.920627 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92329da0f9ec4d2e982efe5e9b94c5d6 has no consensus state
I20240506 19:56:50.921082 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3482045609cb42209a8222185144e10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:50.921664 13249 tablet_bootstrap.cc:492] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:50.921663 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3482045609cb42209a8222185144e10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:50.923323 13249 tablet_bootstrap.cc:654] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.924423 13244 raft_consensus.cc:348] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.924638 13244 raft_consensus.cc:374] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.924702 13244 raft_consensus.cc:729] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.924865 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b3268094cbf481698cb373c1bf6dfba from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:50.924935 13244 consensus_queue.cc:260] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.927989 13244 ts_tablet_manager.cc:1398] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240506 19:56:50.928292 13244 tablet_bootstrap.cc:492] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:50.930336 13244 tablet_bootstrap.cc:654] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:50.936769 13245 tablet_bootstrap.cc:492] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:50.936928 13245 ts_tablet_manager.cc:1367] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.002s
I20240506 19:56:50.937633 13245 raft_consensus.cc:348] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.937834 13245 raft_consensus.cc:374] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:50.937896 13245 raft_consensus.cc:729] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:50.872944 13243 ts_tablet_manager.cc:1888] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240506 19:56:50.939282 13243 log.cc:1196] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/41faccd54f3e49ddb48ea5234f6cdab5
I20240506 19:56:50.939749 13243 ts_tablet_manager.cc:1909] T 41faccd54f3e49ddb48ea5234f6cdab5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:50.940881 13243 tablet_replica.cc:330] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:50.941123 13243 raft_consensus.cc:2229] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.941359 13243 raft_consensus.cc:2258] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.941959 13243 ts_tablet_manager.cc:1875] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:50.933998 13242 ts_tablet_manager.cc:1888] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:50.946048 13242 log.cc:1196] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2519c03738864f6ab5dcfab7bc3fc8aa
I20240506 19:56:50.946406 13242 ts_tablet_manager.cc:1909] T 2519c03738864f6ab5dcfab7bc3fc8aa P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:50.946884 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 691) took 3399 ms. Trace:
I20240506 19:56:50.946978 13242 rpcz_store.cc:276] 0506 19:56:47.547960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.935551 (+387591us) service_pool.cc:225] Handling call
0506 19:56:50.883508 (+2947957us) ts_tablet_manager.cc:1120] Deleting tablet 2519c03738864f6ab5dcfab7bc3fc8aa
0506 19:56:50.883517 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.904688 (+ 21171us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.927020 (+ 22332us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:50.946862 (+ 19842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2946417}
I20240506 19:56:50.947456 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2519c03738864f6ab5dcfab7bc3fc8aa (table default.Table_23d6ef6eb15d489e813fdd5562e5be63 [id=5d61a15249c74e509342122448b76500]) successfully deleted
I20240506 19:56:50.947890 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 687) took 3616 ms. Trace:
I20240506 19:56:50.947973 13242 rpcz_store.cc:276] 0506 19:56:47.331510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:47.933278 (+601768us) service_pool.cc:225] Handling call
0506 19:56:50.947760 (+3014482us) ts_tablet_manager.cc:1120] Deleting tablet 2c96f014293242be9640515697c0733b
0506 19:56:50.947770 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.947873 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005243}
W20240506 19:56:50.949959 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2c96f014293242be9640515697c0733b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c96f014293242be9640515697c0733b already in progress: creating tablet
I20240506 19:56:50.950584 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 704) took 2909 ms. Trace:
I20240506 19:56:50.950714 13242 rpcz_store.cc:276] 0506 19:56:48.041351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.053173 (+ 11822us) service_pool.cc:225] Handling call
0506 19:56:50.950285 (+2897112us) ts_tablet_manager.cc:1120] Deleting tablet cc22261ab2034e818834ba97c2760dad
0506 19:56:50.950293 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.950566 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896820}
W20240506 19:56:50.951319 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet cc22261ab2034e818834ba97c2760dad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc22261ab2034e818834ba97c2760dad already in progress: creating tablet
I20240506 19:56:50.951756 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 705) took 2902 ms. Trace:
I20240506 19:56:50.951872 13242 rpcz_store.cc:276] 0506 19:56:48.049538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.053478 (+ 3940us) service_pool.cc:225] Handling call
0506 19:56:50.951634 (+2898156us) ts_tablet_manager.cc:1120] Deleting tablet 8eb9296bb1104b4cbe279eecd3683796
0506 19:56:50.951657 (+ 23us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.951738 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2898034}
W20240506 19:56:50.952308 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8eb9296bb1104b4cbe279eecd3683796 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb9296bb1104b4cbe279eecd3683796 already in progress: creating tablet
I20240506 19:56:50.952857 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f56d2180b1472a8168fb3a63589065 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.952701 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 706) took 2903 ms. Trace:
I20240506 19:56:50.953191 13242 rpcz_store.cc:276] 0506 19:56:48.049690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.053603 (+ 3913us) service_pool.cc:225] Handling call
0506 19:56:50.952589 (+2898986us) ts_tablet_manager.cc:1120] Deleting tablet df04fb90230d4bf4a060db993adafd14
0506 19:56:50.952597 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.952686 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2898898}
W20240506 19:56:50.953595 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet df04fb90230d4bf4a060db993adafd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df04fb90230d4bf4a060db993adafd14 already in progress: creating tablet
I20240506 19:56:50.953639 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet b899fb38bfed4e02b3708f6abec80e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:50.953909 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 702) took 2985 ms. Trace:
I20240506 19:56:50.953991 13242 rpcz_store.cc:276] 0506 19:56:47.968409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.030424 (+ 62015us) service_pool.cc:225] Handling call
0506 19:56:50.953817 (+2923393us) ts_tablet_manager.cc:1120] Deleting tablet 2a8e331dc24e4d06bf81c8a4b7d70156
0506 19:56:50.953825 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.953895 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871239}
I20240506 19:56:50.954077 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]) successfully deleted
W20240506 19:56:50.954368 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2a8e331dc24e4d06bf81c8a4b7d70156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a8e331dc24e4d06bf81c8a4b7d70156 already in progress: creating tablet
I20240506 19:56:50.954926 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 714) took 2626 ms. Trace:
I20240506 19:56:50.955032 13242 rpcz_store.cc:276] 0506 19:56:48.328980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.329235 (+ 255us) service_pool.cc:225] Handling call
0506 19:56:50.954835 (+2625600us) ts_tablet_manager.cc:1120] Deleting tablet bd3130bc41084bfaabaaecc0985e378d
0506 19:56:50.954843 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.954912 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2625382}
W20240506 19:56:50.955444 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bd3130bc41084bfaabaaecc0985e378d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd3130bc41084bfaabaaecc0985e378d already in progress: creating tablet
I20240506 19:56:50.955799 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 716) took 2573 ms. Trace:
I20240506 19:56:50.955937 13242 rpcz_store.cc:276] 0506 19:56:48.382580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.382816 (+ 236us) service_pool.cc:225] Handling call
0506 19:56:50.955710 (+2572894us) ts_tablet_manager.cc:1120] Deleting tablet 7763959136c44674888916321bd6da24
0506 19:56:50.955718 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.955786 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2571812}
W20240506 19:56:50.956363 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7763959136c44674888916321bd6da24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7763959136c44674888916321bd6da24 already in progress: creating tablet
I20240506 19:56:50.956733 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 715) took 2592 ms. Trace:
I20240506 19:56:50.956847 13242 rpcz_store.cc:276] 0506 19:56:48.364554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.368818 (+ 4264us) service_pool.cc:225] Handling call
0506 19:56:50.956628 (+2587810us) ts_tablet_manager.cc:1120] Deleting tablet 574dbbe8a9a94cd7944b652a68b77403
0506 19:56:50.956647 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.956717 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506540}
I20240506 19:56:50.957515 13242 tablet_replica.cc:330] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:50.957880 13242 raft_consensus.cc:2229] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:50.958257 13242 raft_consensus.cc:2258] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:50.959149 13242 ts_tablet_manager.cc:1875] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:50.957283 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 574dbbe8a9a94cd7944b652a68b77403 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 574dbbe8a9a94cd7944b652a68b77403 already in progress: creating tablet
I20240506 19:56:50.963882 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 104337875517445f862d82ece3a84889 with delete_type TABLET_DATA_DELETED (replaced by tablet 176c61208b1c4518820a829d9f5c0a6f) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:50.966477 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcb357a7eb2147a79cc69d024bd35269 consensus state has no leader
I20240506 19:56:50.968477 14270 raft_consensus.cc:482] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.968647 14270 raft_consensus.cc:504] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.969090 14069 raft_consensus.cc:482] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.969260 14069 raft_consensus.cc:504] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:50.969722 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c63d8995346d78633154a8f0d4c2b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:50.969939 14069 leader_election.cc:288] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:50.970268 13058 leader_election.cc:341] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:50.971076 12781 leader_election.cc:341] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:50.971073 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c63d8995346d78633154a8f0d4c2b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:50.971249 14270 leader_election.cc:288] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:50.971256 12783 leader_election.cc:341] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:50.971350 12783 leader_election.cc:302] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:50.971572 13057 leader_election.cc:341] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.971675 13057 leader_election.cc:302] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:50.971726 14069 raft_consensus.cc:2726] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.972040 14270 raft_consensus.cc:2726] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:50.942682 14285 tablet.cc:1722] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:50.975100 13245 consensus_queue.cc:260] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.975275 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c93117bdcf44c719c2e3588de538e96 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:50.984266 12848 raft_consensus.cc:3037] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:50.988092 13245 ts_tablet_manager.cc:1398] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20240506 19:56:50.990228 12518 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_ecb75e9b5c5d4c4ebd6f3684a6a350f9 [id=034d7824127b4f94aa24bdeb76bfbd7c] alter complete (version 4)
I20240506 19:56:50.994992 14080 raft_consensus.cc:482] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:50.995179 14080 raft_consensus.cc:504] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:50.995860 14080 leader_election.cc:288] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:50.996371 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899fb38bfed4e02b3708f6abec80e0a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:50.997004 12921 leader_election.cc:341] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:50.998854 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899fb38bfed4e02b3708f6abec80e0a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:50.999096 13120 raft_consensus.cc:2454] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:51.000005 12922 leader_election.cc:302] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.001336 14250 raft_consensus.cc:2781] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.001412 14250 raft_consensus.cc:482] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.001470 14250 raft_consensus.cc:3037] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.004079 13242 ts_tablet_manager.cc:1888] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.004204 13242 log.cc:1196] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/0e2005ef58fd4fcea16abe771ee385f8
I20240506 19:56:51.004501 13242 ts_tablet_manager.cc:1909] T 0e2005ef58fd4fcea16abe771ee385f8 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.004844 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 718) took 2429 ms. Trace:
I20240506 19:56:51.004926 13242 rpcz_store.cc:276] 0506 19:56:48.575706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.575755 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:50.957449 (+2381694us) ts_tablet_manager.cc:1120] Deleting tablet 0e2005ef58fd4fcea16abe771ee385f8
0506 19:56:50.957457 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:50.972545 (+ 15088us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.003948 (+ 31403us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.004822 (+ 874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381480}
I20240506 19:56:51.009656 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5ff6037fa555448c912227cbd9c6280f" } new_table_name: "default.Table_f0624408bd8a4b41bccc73648b150210" modify_external_catalogs: true
I20240506 19:56:51.004786 14250 raft_consensus.cc:504] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.011873 14250 leader_election.cc:288] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.012391 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899fb38bfed4e02b3708f6abec80e0a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:51.012974 12921 leader_election.cc:341] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.014065 13245 tablet_bootstrap.cc:492] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.014297 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 0e2005ef58fd4fcea16abe771ee385f8 (table default.Table_577b201c2bf345bdbaeb7bda56b8296c [id=0772215de50d467585bec1f234187f97]) successfully deleted
I20240506 19:56:51.015089 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899fb38bfed4e02b3708f6abec80e0a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:51.015287 13120 raft_consensus.cc:3037] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.015594 13245 tablet_bootstrap.cc:654] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.015602 13242 tablet_replica.cc:330] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:51.015801 13242 raft_consensus.cc:2229] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.015969 13242 raft_consensus.cc:2258] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.017992 13120 raft_consensus.cc:2454] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:51.018307 14288 tablet.cc:1722] T 4e2c9f4baf55412d9bc00818a00c652d P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:51.019946 12922 leader_election.cc:302] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.020501 14250 raft_consensus.cc:2781] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.020669 14250 raft_consensus.cc:686] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:51.020965 14250 consensus_queue.cc:237] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.021507 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_ecb75e9b5c5d4c4ebd6f3684a6a350f9 [id=034d7824127b4f94aa24bdeb76bfbd7c]). Expected version 4 got 0
W20240506 19:56:51.023569 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b1f0e4245774ccab0aa9c6c2ba49c75 consensus state has no leader
I20240506 19:56:51.028350 13244 tablet_bootstrap.cc:492] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:51.028515 13244 ts_tablet_manager.cc:1367] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.013s
I20240506 19:56:51.031282 13244 raft_consensus.cc:348] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.032007 13244 raft_consensus.cc:374] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.032107 13244 raft_consensus.cc:729] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.032327 13244 consensus_queue.cc:260] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.032927 13244 ts_tablet_manager.cc:1398] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:51.036106 13242 ts_tablet_manager.cc:1875] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.038823 14250 raft_consensus.cc:482] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.039057 14250 raft_consensus.cc:504] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.040199 14250 leader_election.cc:288] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.040678 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb90bf767c94d73ace6411cfd31d65c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:51.040964 13120 raft_consensus.cc:2454] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:51.041705 12922 leader_election.cc:302] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:51.042213 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_90043eda47664c8fa37013b6730e5fb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.043247 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_012b93e45a4945ee964f19ccd6bbfa0d" } modify_external_catalogs: true
I20240506 19:56:51.043435 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_012b93e45a4945ee964f19ccd6bbfa0d" } modify_external_catalogs: true
I20240506 19:56:51.044095 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2a11d75c7230425387b56ecf4c87a109" } modify_external_catalogs: true
I20240506 19:56:51.044250 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2a11d75c7230425387b56ecf4c87a109" } modify_external_catalogs: true
I20240506 19:56:51.044335 13244 tablet_bootstrap.cc:492] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:51.045924 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a3a24c46afbb4a6b88e13ace0af3c39b" } modify_external_catalogs: true
I20240506 19:56:51.046180 13244 tablet_bootstrap.cc:654] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.046135 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a3a24c46afbb4a6b88e13ace0af3c39b" } modify_external_catalogs: true
I20240506 19:56:51.046806 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8c81322d630a455595f418e3a97b441e" } modify_external_catalogs: true
I20240506 19:56:51.046968 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8c81322d630a455595f418e3a97b441e" } modify_external_catalogs: true
I20240506 19:56:51.047187 13245 tablet_bootstrap.cc:492] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.047336 13245 ts_tablet_manager.cc:1367] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20240506 19:56:51.048045 13245 raft_consensus.cc:348] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.048235 13245 raft_consensus.cc:374] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.048285 13245 raft_consensus.cc:729] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.048708 13245 consensus_queue.cc:260] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.049264 13245 ts_tablet_manager.cc:1398] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:51.049878 13245 tablet_bootstrap.cc:492] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.051259 13245 tablet_bootstrap.cc:654] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.051534 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_715977f202b24ca7a07c82ef612e23bb" } modify_external_catalogs: true
I20240506 19:56:51.051909 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_715977f202b24ca7a07c82ef612e23bb" } modify_external_catalogs: true
I20240506 19:56:51.054767 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb90bf767c94d73ace6411cfd31d65c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.055158 14250 raft_consensus.cc:2781] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.055410 14250 raft_consensus.cc:482] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.055610 14250 raft_consensus.cc:3037] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:51.056202 12921 leader_election.cc:341] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.060957 13245 tablet_bootstrap.cc:492] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.061012 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_a495d810a1ce4d3a80b146059120ad17 [id=16aa3036ed744751afd08515bb792ce2]). Expected version 1 got 0
I20240506 19:56:51.062279 13245 ts_tablet_manager.cc:1367] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240506 19:56:51.063051 13245 raft_consensus.cc:348] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.063815 13245 raft_consensus.cc:374] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.064064 13245 raft_consensus.cc:729] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.064467 13245 consensus_queue.cc:260] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.065740 13245 ts_tablet_manager.cc:1398] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240506 19:56:51.067620 13245 tablet_bootstrap.cc:492] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.065186 13246 raft_consensus.cc:2258] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.070310 13246 ts_tablet_manager.cc:1875] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.071276 13245 tablet_bootstrap.cc:654] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.072367 14069 raft_consensus.cc:482] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.072539 14069 raft_consensus.cc:504] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.073217 14069 leader_election.cc:288] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:51.074059 12783 leader_election.cc:341] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a8b3ed0b56aa478abb44347afca8fcf6
I20240506 19:56:51.080127 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 88dba275893f4696a5318ec21f0b2477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:35 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.080427 13245 tablet_bootstrap.cc:492] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.081784 13245 ts_tablet_manager.cc:1367] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
W20240506 19:56:51.082022 12781 leader_election.cc:341] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a8b3ed0b56aa478abb44347afca8fcf6
I20240506 19:56:51.082155 12781 leader_election.cc:302] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:51.082576 14069 raft_consensus.cc:2726] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.087837 12515 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 52e6de2ac4744b07a522366d39f4e05a
I20240506 19:56:51.083262 13245 raft_consensus.cc:348] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.091619 13245 raft_consensus.cc:374] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.091712 13245 raft_consensus.cc:729] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.092630 13245 consensus_queue.cc:260] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.092623 14250 raft_consensus.cc:504] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.093313 13245 ts_tablet_manager.cc:1398] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240506 19:56:51.093609 13245 tablet_bootstrap.cc:492] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.094923 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb90bf767c94d73ace6411cfd31d65c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:51.095135 13245 tablet_bootstrap.cc:654] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:51.095614 12921 leader_election.cc:341] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.096141 13246 ts_tablet_manager.cc:1888] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.096423 13246 log.cc:1196] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/4095f9fc28ff48f68064ec8318ff7b51
I20240506 19:56:51.096760 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb90bf767c94d73ace6411cfd31d65c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:51.096844 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90043eda47664c8fa37013b6730e5fb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.096940 13120 raft_consensus.cc:3037] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.099573 13120 raft_consensus.cc:2454] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:51.100484 12922 leader_election.cc:302] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.100842 14080 raft_consensus.cc:2781] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.100956 14080 raft_consensus.cc:686] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:51.101207 14080 consensus_queue.cc:237] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.105899 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:51.106104 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 43460a21f28c41fdb23a6d7f2e942837 (DEFAULT_TABLE table=default.Table_86e7dc4a07f542679409dcc2a3187f46 [id=cdb79dcc2a134fb788f6f1e285ac593d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.106559 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43460a21f28c41fdb23a6d7f2e942837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.108788 13249 tablet_bootstrap.cc:492] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.109691 13249 ts_tablet_manager.cc:1367] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.188s user 0.015s sys 0.020s
I20240506 19:56:51.110230 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 778075705b0b4f00a3184f42b68d2323 (DEFAULT_TABLE table=default.Table_951df9b3f85e416893c271dc056e57ee [id=27ca3baadf324fbe92ecfe455e90428a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.110373 12948 tablet_service.cc:1467] Processing CreateTablet for tablet fd82a7a4a5b54a98b6b0709821d8d922 (DEFAULT_TABLE table=default.Table_113b557a89234f5696e404ae3a3cc0e5 [id=dd3ff8db260242beba43003f552b628c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.110635 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778075705b0b4f00a3184f42b68d2323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.110838 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd82a7a4a5b54a98b6b0709821d8d922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.110810 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 797c3629f1b0413eb471cc899f54b32d (DEFAULT_TABLE table=default.Table_0f8f0a413eab460aa84362ac194994e1 [id=33dfa30213944d2883cefa6f0c6f9365]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.111093 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797c3629f1b0413eb471cc899f54b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.111191 12953 tablet_service.cc:1467] Processing CreateTablet for tablet c650b7f9409742d989159ce3d6599080 (DEFAULT_TABLE table=default.Table_c39e942c7b1948ce8b2d1e47289bcbd7 [id=a33c46e7c94c466394d4a001735cf479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.111568 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c650b7f9409742d989159ce3d6599080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.111629 13249 raft_consensus.cc:348] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.111799 13249 raft_consensus.cc:374] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.111853 13249 raft_consensus.cc:729] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.112053 13249 consensus_queue.cc:260] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.113546 13249 ts_tablet_manager.cc:1398] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:51.113521 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc9e005eb0a4e518b182b219d5771ed (DEFAULT_TABLE table=default.Table_f83a7797a045469bbc2a87de8b086a2a [id=075c32cd3d9f4029be393b323b001186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.113580 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 211497cd9802473da3f4ea40b40830c6 (DEFAULT_TABLE table=default.Table_204da63924eb46ad9044cbc009296da3 [id=537838979f76429da504e67aa7485fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.113827 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc9e005eb0a4e518b182b219d5771ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.114050 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 62f7ed8e26b949028372656f2bbcc8f6 (DEFAULT_TABLE table=default.Table_0b1a8466d287416ab4ec1f38a4687e69 [id=a7fd74ddf9f24e12bdb816bdd9c65fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.114358 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f7ed8e26b949028372656f2bbcc8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.117547 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 691e312510a440898a58bec43ab26e8f (DEFAULT_TABLE table=default.Table_07942e78a30c4f24b8a56a39c0f39e4c [id=ac52c9eb2d9940198e0c4181e2ec793b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.118526 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e312510a440898a58bec43ab26e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.120303 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 797c3629f1b0413eb471cc899f54b32d (DEFAULT_TABLE table=default.Table_0f8f0a413eab460aa84362ac194994e1 [id=33dfa30213944d2883cefa6f0c6f9365]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.120393 12825 tablet_service.cc:1467] Processing CreateTablet for tablet c650b7f9409742d989159ce3d6599080 (DEFAULT_TABLE table=default.Table_c39e942c7b1948ce8b2d1e47289bcbd7 [id=a33c46e7c94c466394d4a001735cf479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.120750 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797c3629f1b0413eb471cc899f54b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.121066 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c650b7f9409742d989159ce3d6599080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.121268 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 778075705b0b4f00a3184f42b68d2323 (DEFAULT_TABLE table=default.Table_951df9b3f85e416893c271dc056e57ee [id=27ca3baadf324fbe92ecfe455e90428a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.121593 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778075705b0b4f00a3184f42b68d2323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.125407 13246 ts_tablet_manager.cc:1909] T 4095f9fc28ff48f68064ec8318ff7b51 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.125598 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 211497cd9802473da3f4ea40b40830c6 (DEFAULT_TABLE table=default.Table_204da63924eb46ad9044cbc009296da3 [id=537838979f76429da504e67aa7485fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.125941 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211497cd9802473da3f4ea40b40830c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.127210 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 751) took 2706 ms. Trace:
I20240506 19:56:51.127298 13246 rpcz_store.cc:276] 0506 19:56:48.420916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.421138 (+ 222us) service_pool.cc:225] Handling call
0506 19:56:50.880147 (+2459009us) ts_tablet_manager.cc:1120] Deleting tablet 4095f9fc28ff48f68064ec8318ff7b51
0506 19:56:50.880155 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.082012 (+201857us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.095761 (+ 13749us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.127187 (+ 31426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458663}
I20240506 19:56:51.127619 13246 tablet_replica.cc:330] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.127804 13246 raft_consensus.cc:2229] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.127936 13246 raft_consensus.cc:2258] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.129578 12827 tablet_service.cc:1467] Processing CreateTablet for tablet fd82a7a4a5b54a98b6b0709821d8d922 (DEFAULT_TABLE table=default.Table_113b557a89234f5696e404ae3a3cc0e5 [id=dd3ff8db260242beba43003f552b628c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.129925 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd82a7a4a5b54a98b6b0709821d8d922. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.130017 12921 consensus_peers.cc:485] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:51.130163 13246 ts_tablet_manager.cc:1875] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.132124 13249 tablet_bootstrap.cc:492] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.132900 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 62f7ed8e26b949028372656f2bbcc8f6 (DEFAULT_TABLE table=default.Table_0b1a8466d287416ab4ec1f38a4687e69 [id=a7fd74ddf9f24e12bdb816bdd9c65fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.133219 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f7ed8e26b949028372656f2bbcc8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.133747 13245 tablet_bootstrap.cc:492] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.133864 13245 ts_tablet_manager.cc:1367] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20240506 19:56:51.134502 13245 raft_consensus.cc:348] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.134681 13245 raft_consensus.cc:374] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.134742 13245 raft_consensus.cc:729] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.134945 13245 consensus_queue.cc:260] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.136132 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 691e312510a440898a58bec43ab26e8f (DEFAULT_TABLE table=default.Table_07942e78a30c4f24b8a56a39c0f39e4c [id=ac52c9eb2d9940198e0c4181e2ec793b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.136463 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e312510a440898a58bec43ab26e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.138959 14080 raft_consensus.cc:482] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.139093 14080 raft_consensus.cc:504] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.139580 13245 ts_tablet_manager.cc:1398] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:51.139609 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc9e005eb0a4e518b182b219d5771ed (DEFAULT_TABLE table=default.Table_f83a7797a045469bbc2a87de8b086a2a [id=075c32cd3d9f4029be393b323b001186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.139914 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc9e005eb0a4e518b182b219d5771ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.140465 13249 tablet_bootstrap.cc:654] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.141953 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176c61208b1c4518820a829d9f5c0a6f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:51.142444 12921 leader_election.cc:341] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.143154 14080 leader_election.cc:288] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.143306 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176c61208b1c4518820a829d9f5c0a6f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:51.143520 13120 raft_consensus.cc:2454] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:51.143919 12922 leader_election.cc:302] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.144311 14080 raft_consensus.cc:2781] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.144351 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211497cd9802473da3f4ea40b40830c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.144378 14080 raft_consensus.cc:482] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.144439 14080 raft_consensus.cc:3037] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:51.145476 12509 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet fd1f00e8620640b08cc054c043900738 (table default.Table_a0ca5a4363dd40c6840fe7989c885cb0 [id=7249e39100454579b86f755187e5aeaa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd1f00e8620640b08cc054c043900738 consensus state has no leader
I20240506 19:56:51.146564 13245 tablet_bootstrap.cc:492] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.147202 14080 raft_consensus.cc:504] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.148368 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176c61208b1c4518820a829d9f5c0a6f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:51.148872 12921 leader_election.cc:341] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.149603 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176c61208b1c4518820a829d9f5c0a6f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:51.149945 13120 raft_consensus.cc:3037] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.150774 13249 tablet_bootstrap.cc:492] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.151003 13249 ts_tablet_manager.cc:1367] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.004s
I20240506 19:56:51.152091 13249 raft_consensus.cc:348] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.152338 13249 raft_consensus.cc:374] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.152448 13249 raft_consensus.cc:729] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.152536 13120 raft_consensus.cc:2454] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:51.152776 13249 consensus_queue.cc:260] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.153012 12922 leader_election.cc:302] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.153138 14080 leader_election.cc:288] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.153568 14274 raft_consensus.cc:2781] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.153681 14274 raft_consensus.cc:686] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:51.153905 14274 consensus_queue.cc:237] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.155392 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 92329da0f9ec4d2e982efe5e9b94c5d6 (DEFAULT_TABLE table=default.Table_8c32c6726a7f4aa2867ec4f2594083d5 [id=bc316585ccd5468cba6c1016998d5d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.155807 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92329da0f9ec4d2e982efe5e9b94c5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.156147 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 0b90b1433a0b4cdfa53316f69ff19085 (DEFAULT_TABLE table=default.Table_7ecd2c36503246d8992d73e00cf06129 [id=e96f8b4e07a6461095da74b4e260046b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.156462 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b90b1433a0b4cdfa53316f69ff19085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.156392 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 65b77b8c5c1a4737b555898ae642a578 (DEFAULT_TABLE table=default.Table_eebcdd01355340739cb45d1acca7daa6 [id=ff67c807ba92484dbda8d03d7df13c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.156723 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b77b8c5c1a4737b555898ae642a578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.156996 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 25cb60a953aa486fac4a985ff0fe2441 (DEFAULT_TABLE table=default.Table_014acbfce7444396ad12f76368ca683f [id=767292be27b246929ffa0f7244be6ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.157445 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cb60a953aa486fac4a985ff0fe2441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.157670 12814 tablet_service.cc:1467] Processing CreateTablet for tablet b92d006097bc4b5ca51a3a8493d75655 (DEFAULT_TABLE table=default.Table_1c043170c0ff4721aeb47b27867693f0 [id=824ad67c294d42f181701cbf63b635c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.157969 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92d006097bc4b5ca51a3a8493d75655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.160691 12828 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4bc6e0c184237bf77ca7feac3b91b (DEFAULT_TABLE table=default.Table_27ea3728f32d484faad6006e1746dad4 [id=99b2a060bcd047588fa6d5e9756ccbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.161010 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4bc6e0c184237bf77ca7feac3b91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.163862 13246 ts_tablet_manager.cc:1888] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.163976 13246 log.cc:1196] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/17fb091762144f6591152680a6acd29e
I20240506 19:56:51.163978 12813 tablet_service.cc:1467] Processing CreateTablet for tablet dbb1c433315c4bdbac4419ef8345c075 (DEFAULT_TABLE table=default.Table_a94082868b214538890b435c8c101305 [id=d2cce1c843f04e4e84256b94e60207b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.164273 13246 ts_tablet_manager.cc:1909] T 17fb091762144f6591152680a6acd29e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.164314 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb1c433315c4bdbac4419ef8345c075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.165040 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 752) took 2743 ms. Trace:
I20240506 19:56:51.165120 13246 rpcz_store.cc:276] 0506 19:56:48.421086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.421340 (+ 254us) service_pool.cc:225] Handling call
0506 19:56:51.127571 (+2706231us) ts_tablet_manager.cc:1120] Deleting tablet 17fb091762144f6591152680a6acd29e
0506 19:56:51.127583 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.140632 (+ 13049us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.163748 (+ 23116us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.165015 (+ 1267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2706029}
I20240506 19:56:51.165453 13246 tablet_replica.cc:330] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.165629 13246 raft_consensus.cc:2229] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:51.165890 13246 pending_rounds.cc:62] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:51.165921 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2a3291123a1e42af80dbfb578c855618
I20240506 19:56:51.165932 13246 pending_rounds.cc:68] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744077196771328 op_type: NO_OP noop_request { }
I20240506 19:56:51.166054 13246 raft_consensus.cc:2866] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:51.166132 13246 raft_consensus.cc:2258] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.166667 13246 ts_tablet_manager.cc:1875] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.166905 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9c93117bdcf44c719c2e3588de538e96
I20240506 19:56:51.167371 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 9c93117bdcf44c719c2e3588de538e96 deleted and replaced by tablet 8d8a94ecb64b4b64889b3585e64fff5a
I20240506 19:56:51.167315 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09d48fbdb5490bbbf58d0daee40b14 (DEFAULT_TABLE table=default.Table_8c2dd5f0bc85416b9c98a687c245f5b5 [id=62473581d9f442d993eb190d59c90efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.167639 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09d48fbdb5490bbbf58d0daee40b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.167778 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5bbcc40911034f319f489931015ca257
I20240506 19:56:51.169576 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 63decce057244d72b4a812368594fcfb (DEFAULT_TABLE table=default.Table_de7835c60b14419892d8827447c27f2e [id=561add2e304a497da06ce1abc6054980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.169935 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63decce057244d72b4a812368594fcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.170071 12522 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet edf3ccd8324f42949018b0a3b803961c
I20240506 19:56:51.170591 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2b3268094cbf481698cb373c1bf6dfba
I20240506 19:56:51.171100 12513 catalog_manager.cc:6554] ReplaceTablet: tablet 2b3268094cbf481698cb373c1bf6dfba deleted and replaced by tablet 597c61f1d1104256b90f6601cf3e6667
I20240506 19:56:51.173043 12819 tablet_service.cc:1467] Processing CreateTablet for tablet a152a0f6a9ed4aa5b74fe5a53688331d (DEFAULT_TABLE table=default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5 [id=16258fb4a39c4effb6effa8e857b0b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.173404 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a152a0f6a9ed4aa5b74fe5a53688331d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.173544 13246 ts_tablet_manager.cc:1888] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:51.173646 13246 log.cc:1196] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/7de7397db1e74b4488a3f9a70136bccb
I20240506 19:56:51.174002 13246 ts_tablet_manager.cc:1909] T 7de7397db1e74b4488a3f9a70136bccb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.174913 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 753) took 2753 ms. Trace:
I20240506 19:56:51.175021 13246 rpcz_store.cc:276] 0506 19:56:48.421753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.421791 (+ 38us) service_pool.cc:225] Handling call
0506 19:56:51.165408 (+2743617us) ts_tablet_manager.cc:1120] Deleting tablet 7de7397db1e74b4488a3f9a70136bccb
0506 19:56:51.165417 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.169582 (+ 4165us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.173426 (+ 3844us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.174888 (+ 1462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2743474}
I20240506 19:56:51.175359 13246 tablet_replica.cc:330] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.175555 13246 raft_consensus.cc:2229] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:51.175944 13246 raft_consensus.cc:2258] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.176471 13246 ts_tablet_manager.cc:1875] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.176812 12812 tablet_service.cc:1467] Processing CreateTablet for tablet b743d0c18c9149fca7da961dade6ad80 (DEFAULT_TABLE table=default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020 [id=4a250d3001604130bc1862e260086e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.177161 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b743d0c18c9149fca7da961dade6ad80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.179919 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7c15bd15fd9f442a986827cad7197380"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.180379 12810 tablet_service.cc:1467] Processing CreateTablet for tablet dc4da6cfffd74edb8d34019662e430d6 (DEFAULT_TABLE table=default.Table_9f4ce0d827e5469ba860bae2d23245f5 [id=3731ecdeeed84de1b6d350e8283966a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.180773 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc4da6cfffd74edb8d34019662e430d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.181670 13245 tablet_bootstrap.cc:654] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:51.182354 13058 consensus_peers.cc:485] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:51.182395 13246 ts_tablet_manager.cc:1888] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:51.182490 13246 log.cc:1196] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/796628a0859c4511a8cb35e495dc8886
I20240506 19:56:51.182821 13246 ts_tablet_manager.cc:1909] T 796628a0859c4511a8cb35e495dc8886 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:51.183485 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c15bd15fd9f442a986827cad7197380 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.183760 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 754) took 2636 ms. Trace:
I20240506 19:56:51.183864 13246 rpcz_store.cc:276] 0506 19:56:48.547126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.547206 (+ 80us) service_pool.cc:225] Handling call
0506 19:56:51.175306 (+2628100us) ts_tablet_manager.cc:1120] Deleting tablet 796628a0859c4511a8cb35e495dc8886
0506 19:56:51.175316 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.179500 (+ 4184us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.182303 (+ 2803us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.183737 (+ 1434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2627839}
I20240506 19:56:51.184273 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 755) took 2452 ms. Trace:
I20240506 19:56:51.184346 13246 rpcz_store.cc:276] 0506 19:56:48.731693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.731759 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:51.184154 (+2452395us) ts_tablet_manager.cc:1120] Deleting tablet 8689ec0a8e134bceb010053b1f5a343d
0506 19:56:51.184163 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.184256 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2452155}
I20240506 19:56:51.184681 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 756) took 2413 ms. Trace:
I20240506 19:56:51.184764 13246 rpcz_store.cc:276] 0506 19:56:48.771602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.771733 (+ 131us) service_pool.cc:225] Handling call
0506 19:56:51.184575 (+2412842us) ts_tablet_manager.cc:1120] Deleting tablet 9e17d0a261734d45866525b35de7cba3
0506 19:56:51.184585 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.184665 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2412662}
I20240506 19:56:51.185087 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 757) took 2409 ms. Trace:
I20240506 19:56:51.185166 13246 rpcz_store.cc:276] 0506 19:56:48.776116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.776176 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:51.185000 (+2408824us) ts_tablet_manager.cc:1120] Deleting tablet c5f3def738fb4cb195d08e48d7d2b293
0506 19:56:51.185008 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.185074 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367202}
I20240506 19:56:51.185530 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 758) took 2396 ms. Trace:
I20240506 19:56:51.185614 13246 rpcz_store.cc:276] 0506 19:56:48.789540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.789936 (+ 396us) service_pool.cc:225] Handling call
0506 19:56:51.185438 (+2395502us) ts_tablet_manager.cc:1120] Deleting tablet 8689ec0a8e134bceb010053b1f5a343d
0506 19:56:51.185446 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.185516 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367505}
I20240506 19:56:51.185406 13249 ts_tablet_manager.cc:1398] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20240506 19:56:51.185914 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 762) took 2373 ms. Trace:
I20240506 19:56:51.185981 13246 rpcz_store.cc:276] 0506 19:56:48.812462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.812526 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:51.185822 (+2373296us) ts_tablet_manager.cc:1120] Deleting tablet e1d7a850a3e644c3a38ab224efcc8af5
0506 19:56:51.185830 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.185899 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367474}
I20240506 19:56:51.186583 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09d48fbdb5490bbbf58d0daee40b14 (DEFAULT_TABLE table=default.Table_8c2dd5f0bc85416b9c98a687c245f5b5 [id=62473581d9f442d993eb190d59c90efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.186750 12965 tablet_service.cc:1467] Processing CreateTablet for tablet a152a0f6a9ed4aa5b74fe5a53688331d (DEFAULT_TABLE table=default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5 [id=16258fb4a39c4effb6effa8e857b0b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.187005 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09d48fbdb5490bbbf58d0daee40b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.186975 12959 tablet_service.cc:1467] Processing CreateTablet for tablet dc4da6cfffd74edb8d34019662e430d6 (DEFAULT_TABLE table=default.Table_9f4ce0d827e5469ba860bae2d23245f5 [id=3731ecdeeed84de1b6d350e8283966a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.187286 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc4da6cfffd74edb8d34019662e430d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.187263 12955 tablet_service.cc:1467] Processing CreateTablet for tablet b743d0c18c9149fca7da961dade6ad80 (DEFAULT_TABLE table=default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020 [id=4a250d3001604130bc1862e260086e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.187537 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 63decce057244d72b4a812368594fcfb (DEFAULT_TABLE table=default.Table_de7835c60b14419892d8827447c27f2e [id=561add2e304a497da06ce1abc6054980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.187737 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 25cb60a953aa486fac4a985ff0fe2441 (DEFAULT_TABLE table=default.Table_014acbfce7444396ad12f76368ca683f [id=767292be27b246929ffa0f7244be6ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.187558 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b743d0c18c9149fca7da961dade6ad80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.188165 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a152a0f6a9ed4aa5b74fe5a53688331d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.188362 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63decce057244d72b4a812368594fcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.188385 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 64fabc03ad114a80a236a629db6fd311 (DEFAULT_TABLE table=default.Table_7b12e2d9a5d0496586f34ba591615326 [id=fab32d0c08604ab7976f3c3ef48e4859]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.188668 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64fabc03ad114a80a236a629db6fd311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.189010 14223 raft_consensus.cc:482] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.189136 14223 raft_consensus.cc:504] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.189765 14223 leader_election.cc:288] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.191974 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b59ad067c274fddbba5f71e89480dec" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.191972 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b59ad067c274fddbba5f71e89480dec" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:51.192333 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3291123a1e42af80dbfb578c855618 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.192493 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c93117bdcf44c719c2e3588de538e96 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d8a94ecb64b4b64889b3585e64fff5a) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.192649 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbcc40911034f319f489931015ca257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.192798 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3268094cbf481698cb373c1bf6dfba with delete_type TABLET_DATA_DELETED (replaced by tablet 597c61f1d1104256b90f6601cf3e6667) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:51.192824 13057 leader_election.cc:341] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.187737 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 0b90b1433a0b4cdfa53316f69ff19085 (DEFAULT_TABLE table=default.Table_7ecd2c36503246d8992d73e00cf06129 [id=e96f8b4e07a6461095da74b4e260046b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.193222 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b90b1433a0b4cdfa53316f69ff19085. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.193625 13058 leader_election.cc:341] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.193717 13058 leader_election.cc:302] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.188205 12964 tablet_service.cc:1467] Processing CreateTablet for tablet cd244341d9e34bc595f1420f2977907f (DEFAULT_TABLE table=default.Table_e1222c6575614a1d83d72a3751290c35 [id=8ac1522e079245629612384cd7cad952]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.194135 14223 raft_consensus.cc:2726] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.194340 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd244341d9e34bc595f1420f2977907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.194375 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 761) took 2382 ms. Trace:
I20240506 19:56:51.194504 13246 rpcz_store.cc:276] 0506 19:56:48.812373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.812621 (+ 248us) service_pool.cc:225] Handling call
0506 19:56:51.186186 (+2373565us) ts_tablet_manager.cc:1120] Deleting tablet ce6483470c864697be6d9590ec95edfc
0506 19:56:51.186193 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.194353 (+ 8160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367779}
I20240506 19:56:51.194568 12953 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4bc6e0c184237bf77ca7feac3b91b (DEFAULT_TABLE table=default.Table_27ea3728f32d484faad6006e1746dad4 [id=99b2a060bcd047588fa6d5e9756ccbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.195020 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4bc6e0c184237bf77ca7feac3b91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.194993 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 763) took 2382 ms. Trace:
I20240506 19:56:51.195120 13246 rpcz_store.cc:276] 0506 19:56:48.812574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.812759 (+ 185us) service_pool.cc:225] Handling call
0506 19:56:51.194870 (+2382111us) ts_tablet_manager.cc:1120] Deleting tablet a8448ad8e9384edba7202d0bdce516a4
0506 19:56:51.194882 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.194974 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376363}
I20240506 19:56:51.195380 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cb60a953aa486fac4a985ff0fe2441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.195616 13246 tablet_replica.cc:330] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.195873 13246 raft_consensus.cc:2229] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:51.196612 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 211497cd9802473da3f4ea40b40830c6 (DEFAULT_TABLE table=default.Table_204da63924eb46ad9044cbc009296da3 [id=537838979f76429da504e67aa7485fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.197031 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211497cd9802473da3f4ea40b40830c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.197656 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 797c3629f1b0413eb471cc899f54b32d (DEFAULT_TABLE table=default.Table_0f8f0a413eab460aa84362ac194994e1 [id=33dfa30213944d2883cefa6f0c6f9365]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.197971 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797c3629f1b0413eb471cc899f54b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.199633 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 691e312510a440898a58bec43ab26e8f (DEFAULT_TABLE table=default.Table_07942e78a30c4f24b8a56a39c0f39e4c [id=ac52c9eb2d9940198e0c4181e2ec793b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.200033 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e312510a440898a58bec43ab26e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.201565 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 778075705b0b4f00a3184f42b68d2323 (DEFAULT_TABLE table=default.Table_951df9b3f85e416893c271dc056e57ee [id=27ca3baadf324fbe92ecfe455e90428a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.201601 13096 tablet_service.cc:1467] Processing CreateTablet for tablet c650b7f9409742d989159ce3d6599080 (DEFAULT_TABLE table=default.Table_c39e942c7b1948ce8b2d1e47289bcbd7 [id=a33c46e7c94c466394d4a001735cf479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.201869 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778075705b0b4f00a3184f42b68d2323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.202219 13092 tablet_service.cc:1467] Processing CreateTablet for tablet b92d006097bc4b5ca51a3a8493d75655 (DEFAULT_TABLE table=default.Table_1c043170c0ff4721aeb47b27867693f0 [id=824ad67c294d42f181701cbf63b635c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.202518 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92d006097bc4b5ca51a3a8493d75655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.204937 13103 tablet_service.cc:1467] Processing CreateTablet for tablet fd82a7a4a5b54a98b6b0709821d8d922 (DEFAULT_TABLE table=default.Table_113b557a89234f5696e404ae3a3cc0e5 [id=dd3ff8db260242beba43003f552b628c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.205237 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd82a7a4a5b54a98b6b0709821d8d922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.205552 13099 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4bc6e0c184237bf77ca7feac3b91b (DEFAULT_TABLE table=default.Table_27ea3728f32d484faad6006e1746dad4 [id=99b2a060bcd047588fa6d5e9756ccbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.205835 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4bc6e0c184237bf77ca7feac3b91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.210616 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 62f7ed8e26b949028372656f2bbcc8f6 (DEFAULT_TABLE table=default.Table_0b1a8466d287416ab4ec1f38a4687e69 [id=a7fd74ddf9f24e12bdb816bdd9c65fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.210988 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f7ed8e26b949028372656f2bbcc8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.213399 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c650b7f9409742d989159ce3d6599080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.213578 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc9e005eb0a4e518b182b219d5771ed (DEFAULT_TABLE table=default.Table_f83a7797a045469bbc2a87de8b086a2a [id=075c32cd3d9f4029be393b323b001186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.213907 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc9e005eb0a4e518b182b219d5771ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.217587 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 92329da0f9ec4d2e982efe5e9b94c5d6 (DEFAULT_TABLE table=default.Table_8c32c6726a7f4aa2867ec4f2594083d5 [id=bc316585ccd5468cba6c1016998d5d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.217917 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92329da0f9ec4d2e982efe5e9b94c5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.221321 13249 tablet_bootstrap.cc:492] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.221561 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 65b77b8c5c1a4737b555898ae642a578 (DEFAULT_TABLE table=default.Table_eebcdd01355340739cb45d1acca7daa6 [id=ff67c807ba92484dbda8d03d7df13c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.221899 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b77b8c5c1a4737b555898ae642a578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.224615 12822 tablet_service.cc:1467] Processing CreateTablet for tablet cd244341d9e34bc595f1420f2977907f (DEFAULT_TABLE table=default.Table_e1222c6575614a1d83d72a3751290c35 [id=8ac1522e079245629612384cd7cad952]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.225018 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd244341d9e34bc595f1420f2977907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.225088 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 64fabc03ad114a80a236a629db6fd311 (DEFAULT_TABLE table=default.Table_7b12e2d9a5d0496586f34ba591615326 [id=fab32d0c08604ab7976f3c3ef48e4859]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.225562 13100 tablet_service.cc:1467] Processing CreateTablet for tablet dbb1c433315c4bdbac4419ef8345c075 (DEFAULT_TABLE table=default.Table_a94082868b214538890b435c8c101305 [id=d2cce1c843f04e4e84256b94e60207b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.225886 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb1c433315c4bdbac4419ef8345c075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.226316 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3291123a1e42af80dbfb578c855618 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.226531 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c93117bdcf44c719c2e3588de538e96 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d8a94ecb64b4b64889b3585e64fff5a) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.226681 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbcc40911034f319f489931015ca257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.226809 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3268094cbf481698cb373c1bf6dfba with delete_type TABLET_DATA_DELETED (replaced by tablet 597c61f1d1104256b90f6601cf3e6667) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.229583 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09d48fbdb5490bbbf58d0daee40b14 (DEFAULT_TABLE table=default.Table_8c2dd5f0bc85416b9c98a687c245f5b5 [id=62473581d9f442d993eb190d59c90efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.229916 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09d48fbdb5490bbbf58d0daee40b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.196247 13246 pending_rounds.cc:62] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:51.232532 13246 pending_rounds.cc:68] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744078201184256 op_type: NO_OP noop_request { }
I20240506 19:56:51.232709 13246 raft_consensus.cc:2866] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:51.233072 13246 raft_consensus.cc:2258] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.233569 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 63decce057244d72b4a812368594fcfb (DEFAULT_TABLE table=default.Table_de7835c60b14419892d8827447c27f2e [id=561add2e304a497da06ce1abc6054980]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.233779 13246 ts_tablet_manager.cc:1875] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.233913 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63decce057244d72b4a812368594fcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.236753 14080 raft_consensus.cc:482] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.236972 14080 raft_consensus.cc:504] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.238126 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e8b8be953a4a5784f2d098fd6364e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.238766 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 4095f9fc28ff48f68064ec8318ff7b51 (table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]) successfully deleted
W20240506 19:56:51.238878 12921 leader_election.cc:341] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.239010 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 17fb091762144f6591152680a6acd29e (table default.Table_82d6bee639334fca838d2adf031efd21 [id=d8e5bdbac7204b04ba040114b49d6ee8]) successfully deleted
I20240506 19:56:51.239246 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 7de7397db1e74b4488a3f9a70136bccb (table default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]) successfully deleted
I20240506 19:56:51.239269 14223 raft_consensus.cc:482] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.239434 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]) successfully deleted
I20240506 19:56:51.239390 14223 raft_consensus.cc:504] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:51.239653 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8689ec0a8e134bceb010053b1f5a343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8689ec0a8e134bceb010053b1f5a343d already in progress: creating tablet
I20240506 19:56:51.239881 14080 leader_election.cc:288] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:51.239903 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9e17d0a261734d45866525b35de7cba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e17d0a261734d45866525b35de7cba3 already in progress: creating tablet
I20240506 19:56:51.240015 14223 leader_election.cc:288] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.240494 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47449a5a504f939e131743d94433ee" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:51.240576 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e8b8be953a4a5784f2d098fd6364e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:51.240725 13120 raft_consensus.cc:2454] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:51.240973 13057 leader_election.cc:341] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:51.240685 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c5f3def738fb4cb195d08e48d7d2b293 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5f3def738fb4cb195d08e48d7d2b293 already in progress: creating tablet
I20240506 19:56:51.241215 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47449a5a504f939e131743d94433ee" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.241423 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64fabc03ad114a80a236a629db6fd311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.242302 13249 tablet_bootstrap.cc:654] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:51.244310 13058 leader_election.cc:341] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.244441 13058 leader_election.cc:302] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.244779 14223 raft_consensus.cc:2726] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.245380 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 43460a21f28c41fdb23a6d7f2e942837 (DEFAULT_TABLE table=default.Table_86e7dc4a07f542679409dcc2a3187f46 [id=cdb79dcc2a134fb788f6f1e285ac593d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.245882 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43460a21f28c41fdb23a6d7f2e942837. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.246021 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8689ec0a8e134bceb010053b1f5a343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8689ec0a8e134bceb010053b1f5a343d already in progress: creating tablet
W20240506 19:56:51.246333 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet e1d7a850a3e644c3a38ab224efcc8af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d7a850a3e644c3a38ab224efcc8af5 already in progress: creating tablet
I20240506 19:56:51.244938 12922 leader_election.cc:302] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.245276 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 43460a21f28c41fdb23a6d7f2e942837 (DEFAULT_TABLE table=default.Table_86e7dc4a07f542679409dcc2a3187f46 [id=cdb79dcc2a134fb788f6f1e285ac593d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.247375 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43460a21f28c41fdb23a6d7f2e942837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.247779 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 70b6a622cd9e483ca707440b4759e48f (DEFAULT_TABLE table=default.Table_f00049dcfdce4881891eed1bf5951d5b [id=f2b74839f0a0480f8d37f0d11f3743ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.248208 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b6a622cd9e483ca707440b4759e48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.251439 13242 ts_tablet_manager.cc:1888] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:51.251562 13242 log.cc:1196] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/796628a0859c4511a8cb35e495dc8886
I20240506 19:56:51.251901 13242 ts_tablet_manager.cc:1909] T 796628a0859c4511a8cb35e495dc8886 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.252717 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 717) took 2677 ms. Trace:
I20240506 19:56:51.252810 13242 rpcz_store.cc:276] 0506 19:56:48.575558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.576010 (+ 452us) service_pool.cc:225] Handling call
0506 19:56:51.015541 (+2439531us) ts_tablet_manager.cc:1120] Deleting tablet 796628a0859c4511a8cb35e495dc8886
0506 19:56:51.015555 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.039950 (+ 24395us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.251344 (+211394us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.252695 (+ 1351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2439281}
I20240506 19:56:51.253003 13104 tablet_service.cc:1467] Processing CreateTablet for tablet a152a0f6a9ed4aa5b74fe5a53688331d (DEFAULT_TABLE table=default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5 [id=16258fb4a39c4effb6effa8e857b0b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.253181 13242 tablet_replica.cc:330] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:51.246565 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ce6483470c864697be6d9590ec95edfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce6483470c864697be6d9590ec95edfc already in progress: creating tablet
I20240506 19:56:51.253409 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a152a0f6a9ed4aa5b74fe5a53688331d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.253440 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 796628a0859c4511a8cb35e495dc8886 (table default.Table_3dea609720914447924d3591bfed24f6 [id=7aa2f61e032f4db29e4bbd9d7505a885]) successfully deleted
W20240506 19:56:51.253636 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a8448ad8e9384edba7202d0bdce516a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8448ad8e9384edba7202d0bdce516a4 already in progress: creating tablet
I20240506 19:56:51.254200 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d7e3f13ea1c1428b9ca232be80f34d5b" } new_table_name: "default.Table_a711b3a1487b4379a9e0eb9dacf05c1d" modify_external_catalogs: true
I20240506 19:56:51.247516 14080 raft_consensus.cc:2781] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.254793 14080 raft_consensus.cc:482] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.254901 14080 raft_consensus.cc:3037] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.257457 14080 raft_consensus.cc:504] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.258550 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e8b8be953a4a5784f2d098fd6364e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:51.259101 12921 leader_election.cc:341] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.263048 14080 leader_election.cc:288] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.265980 13249 tablet_bootstrap.cc:492] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.266253 13242 raft_consensus.cc:2229] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.266520 13242 raft_consensus.cc:2258] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.267156 13242 ts_tablet_manager.cc:1875] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.267284 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2d668c000c934cf7906b877bc0938038"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.268100 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d668c000c934cf7906b877bc0938038 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.268728 13249 ts_tablet_manager.cc:1367] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240506 19:56:51.269831 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 0d29c7c9c933414fb0ca6e6d370eac62 (DEFAULT_TABLE table=default.Table_1b1ba78c0a194d509cee193f3bec562e [id=af9eea916c5f44b2a0c650481873e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.270866 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e8b8be953a4a5784f2d098fd6364e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:51.271066 13120 raft_consensus.cc:3037] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.271003 14249 consensus_queue.cc:1035] T be6e733b6efd4041abf465c7bb788ef5 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:51.269831 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 4531f67f8fb34af5b12ef1c3316f84b9 (DEFAULT_TABLE table=default.Table_e74f7640674041f292cd9a7b4ffb3e9e [id=22080408cdb6434b89a4834e857cf550]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.272027 13246 ts_tablet_manager.cc:1888] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:51.272132 13246 log.cc:1196] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/034d6309de0740d0a5806b3847e334cb
I20240506 19:56:51.272508 13246 ts_tablet_manager.cc:1909] T 034d6309de0740d0a5806b3847e334cb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.272742 13249 raft_consensus.cc:348] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.272923 13249 raft_consensus.cc:374] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.272970 13249 raft_consensus.cc:729] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.273907 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_686d4b3ab7454f2e87e3fab0797c3cee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.274672 13249 consensus_queue.cc:260] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:51.274798 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_686d4b3ab7454f2e87e3fab0797c3cee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.275712 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 767) took 2410 ms. Trace:
I20240506 19:56:51.276584 13249 ts_tablet_manager.cc:1398] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:51.276441 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_00c419ed22644c6cb9dd58b4b5d5ca39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.275831 13246 rpcz_store.cc:276] 0506 19:56:48.862988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.863300 (+ 312us) service_pool.cc:225] Handling call
0506 19:56:51.195565 (+2332265us) ts_tablet_manager.cc:1120] Deleting tablet 034d6309de0740d0a5806b3847e334cb
0506 19:56:51.195576 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.249246 (+ 53670us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.268391 (+ 19145us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.273283 (+ 4892us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332082}
W20240506 19:56:51.276984 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00c419ed22644c6cb9dd58b4b5d5ca39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.277035 13246 tablet_replica.cc:330] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.277222 13246 raft_consensus.cc:2229] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.277366 13249 tablet_bootstrap.cc:492] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.277516 13246 raft_consensus.cc:2258] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.278672 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4343df7e98714672b87cdc89128871a6" } new_table_name: "default.Table_e63955b8a5324e6688d4a2126f9a2315" modify_external_catalogs: true
I20240506 19:56:51.278570 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 5553e49640594e258d3506b9250a6c9f (DEFAULT_TABLE table=default.Table_4343df7e98714672b87cdc89128871a6 [id=5eae429c4d3c4a199359ea1c6532b6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.279646 13249 tablet_bootstrap.cc:654] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.280723 13097 tablet_service.cc:1467] Processing CreateTablet for tablet dc4da6cfffd74edb8d34019662e430d6 (DEFAULT_TABLE table=default.Table_9f4ce0d827e5469ba860bae2d23245f5 [id=3731ecdeeed84de1b6d350e8283966a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.281150 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc4da6cfffd74edb8d34019662e430d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.281484 13246 ts_tablet_manager.cc:1875] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.282361 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 8474271b3e334b50949d160676a61f0e (DEFAULT_TABLE table=default.Table_303d9190978b40a89152b877de75abb4 [id=7fe70815a63a40139593dcf8b27b60bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.286024 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 25cb60a953aa486fac4a985ff0fe2441 (DEFAULT_TABLE table=default.Table_014acbfce7444396ad12f76368ca683f [id=767292be27b246929ffa0f7244be6ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.286427 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cb60a953aa486fac4a985ff0fe2441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.287590 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c2e72287331646f2b30cf2f0e8d0509a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.288283 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2e72287331646f2b30cf2f0e8d0509a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.289804 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbcc40911034f319f489931015ca257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.290181 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 70b6a622cd9e483ca707440b4759e48f (DEFAULT_TABLE table=default.Table_f00049dcfdce4881891eed1bf5951d5b [id=f2b74839f0a0480f8d37f0d11f3743ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.290565 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b6a622cd9e483ca707440b4759e48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.290702 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 034d6309de0740d0a5806b3847e334cb (table default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]) successfully deleted
I20240506 19:56:51.291301 13120 raft_consensus.cc:2454] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:51.293114 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3268094cbf481698cb373c1bf6dfba with delete_type TABLET_DATA_DELETED (replaced by tablet 597c61f1d1104256b90f6601cf3e6667) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.293537 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 4531f67f8fb34af5b12ef1c3316f84b9 (DEFAULT_TABLE table=default.Table_e74f7640674041f292cd9a7b4ffb3e9e [id=22080408cdb6434b89a4834e857cf550]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.293929 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4531f67f8fb34af5b12ef1c3316f84b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.295125 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c93117bdcf44c719c2e3588de538e96 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d8a94ecb64b4b64889b3585e64fff5a) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.296190 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 0d29c7c9c933414fb0ca6e6d370eac62 (DEFAULT_TABLE table=default.Table_1b1ba78c0a194d509cee193f3bec562e [id=af9eea916c5f44b2a0c650481873e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.296566 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d29c7c9c933414fb0ca6e6d370eac62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.296733 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 5553e49640594e258d3506b9250a6c9f (DEFAULT_TABLE table=default.Table_4343df7e98714672b87cdc89128871a6 [id=5eae429c4d3c4a199359ea1c6532b6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.297062 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5553e49640594e258d3506b9250a6c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.299369 12922 leader_election.cc:302] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.299443 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 64fabc03ad114a80a236a629db6fd311 (DEFAULT_TABLE table=default.Table_7b12e2d9a5d0496586f34ba591615326 [id=fab32d0c08604ab7976f3c3ef48e4859]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.299710 14274 raft_consensus.cc:2781] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.299820 14274 raft_consensus.cc:686] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:51.299927 13092 tablet_service.cc:1467] Processing CreateTablet for tablet cd244341d9e34bc595f1420f2977907f (DEFAULT_TABLE table=default.Table_e1222c6575614a1d83d72a3751290c35 [id=8ac1522e079245629612384cd7cad952]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.300076 14274 consensus_queue.cc:237] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.295069 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3291123a1e42af80dbfb578c855618 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.300714 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 8474271b3e334b50949d160676a61f0e (DEFAULT_TABLE table=default.Table_303d9190978b40a89152b877de75abb4 [id=7fe70815a63a40139593dcf8b27b60bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.301023 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:51.313802 13249 tablet_bootstrap.cc:492] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.315963 13249 ts_tablet_manager.cc:1367] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.000s
I20240506 19:56:51.317884 13249 raft_consensus.cc:348] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.325421 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8474271b3e334b50949d160676a61f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.322685 13249 raft_consensus.cc:374] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.327824 13249 raft_consensus.cc:729] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.328505 13249 consensus_queue.cc:260] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.329748 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d29c7c9c933414fb0ca6e6d370eac62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.330199 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4531f67f8fb34af5b12ef1c3316f84b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.188072 12962 tablet_service.cc:1467] Processing CreateTablet for tablet b92d006097bc4b5ca51a3a8493d75655 (DEFAULT_TABLE table=default.Table_1c043170c0ff4721aeb47b27867693f0 [id=824ad67c294d42f181701cbf63b635c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.336516 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5553e49640594e258d3506b9250a6c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.335902 13088 tablet_service.cc:1467] Processing CreateTablet for tablet b743d0c18c9149fca7da961dade6ad80 (DEFAULT_TABLE table=default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020 [id=4a250d3001604130bc1862e260086e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.341655 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 0b90b1433a0b4cdfa53316f69ff19085 (DEFAULT_TABLE table=default.Table_7ecd2c36503246d8992d73e00cf06129 [id=e96f8b4e07a6461095da74b4e260046b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.345223 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8474271b3e334b50949d160676a61f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.349036 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b743d0c18c9149fca7da961dade6ad80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.349412 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd244341d9e34bc595f1420f2977907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.349754 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b90b1433a0b4cdfa53316f69ff19085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.350129 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64fabc03ad114a80a236a629db6fd311. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.366752 13211 log.cc:924] Time spent T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20240506 19:56:51.336696 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92d006097bc4b5ca51a3a8493d75655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.337534 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 65b77b8c5c1a4737b555898ae642a578 (DEFAULT_TABLE table=default.Table_eebcdd01355340739cb45d1acca7daa6 [id=ff67c807ba92484dbda8d03d7df13c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.369886 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b77b8c5c1a4737b555898ae642a578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.370002 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 4531f67f8fb34af5b12ef1c3316f84b9 (DEFAULT_TABLE table=default.Table_e74f7640674041f292cd9a7b4ffb3e9e [id=22080408cdb6434b89a4834e857cf550]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.370335 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4531f67f8fb34af5b12ef1c3316f84b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.378437 14223 raft_consensus.cc:482] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.378605 14223 raft_consensus.cc:504] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.379236 14223 leader_election.cc:288] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.379856 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796beb8e0a2c41d698bd4c05796263f1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.380318 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796beb8e0a2c41d698bd4c05796263f1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:51.380492 13058 leader_election.cc:341] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:51.380831 13057 leader_election.cc:341] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.380939 13057 leader_election.cc:302] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.381098 14250 leader_election.cc:288] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.381192 14223 raft_consensus.cc:2726] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.361367 12952 tablet_service.cc:1467] Processing CreateTablet for tablet dbb1c433315c4bdbac4419ef8345c075 (DEFAULT_TABLE table=default.Table_a94082868b214538890b435c8c101305 [id=d2cce1c843f04e4e84256b94e60207b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.383097 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb1c433315c4bdbac4419ef8345c075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.383738 13243 ts_tablet_manager.cc:1888] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.383850 13243 log.cc:1196] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f8951bf4873e4382bcd42a52368fb376
I20240506 19:56:51.331657 13249 ts_tablet_manager.cc:1398] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240506 19:56:51.384210 13243 ts_tablet_manager.cc:1909] T f8951bf4873e4382bcd42a52368fb376 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:51.363957 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 70b6a622cd9e483ca707440b4759e48f (DEFAULT_TABLE table=default.Table_f00049dcfdce4881891eed1bf5951d5b [id=f2b74839f0a0480f8d37f0d11f3743ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.363957 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 92329da0f9ec4d2e982efe5e9b94c5d6 (DEFAULT_TABLE table=default.Table_8c32c6726a7f4aa2867ec4f2594083d5 [id=bc316585ccd5468cba6c1016998d5d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.385937 13242 ts_tablet_manager.cc:1888] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.386068 13242 log.cc:1196] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ca394b7f53ce445e8778dd25d0cf25c7
I20240506 19:56:51.386336 13242 ts_tablet_manager.cc:1909] T ca394b7f53ce445e8778dd25d0cf25c7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.386621 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 719) took 2780 ms. Trace:
I20240506 19:56:51.386695 13242 rpcz_store.cc:276] 0506 19:56:48.606528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.606645 (+ 117us) service_pool.cc:225] Handling call
0506 19:56:51.253131 (+2646486us) ts_tablet_manager.cc:1120] Deleting tablet ca394b7f53ce445e8778dd25d0cf25c7
0506 19:56:51.253142 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.368805 (+115663us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.382324 (+ 13519us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.386605 (+ 4281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2645031}
I20240506 19:56:51.387174 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ca394b7f53ce445e8778dd25d0cf25c7 (table default.Table_ca61f3a9aefa4fba8fea32d717de3fb1 [id=bf75c039ad324e50a2ccb889439e2b4c]) successfully deleted
I20240506 19:56:51.390045 14199 log.cc:824] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Log is configured to *not* fsync() on all Append() calls
I20240506 19:56:51.400283 13242 tablet_replica.cc:330] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:51.400580 13242 raft_consensus.cc:2229] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.400746 13242 raft_consensus.cc:2258] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.401473 13249 tablet_bootstrap.cc:492] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.402917 13249 tablet_bootstrap.cc:654] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.404809 13242 ts_tablet_manager.cc:1875] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.406380 13246 ts_tablet_manager.cc:1888] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.406519 13246 log.cc:1196] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/bafd3c4706f74e7da36a8f3538760f43
I20240506 19:56:51.409121 13246 ts_tablet_manager.cc:1909] T bafd3c4706f74e7da36a8f3538760f43 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.409914 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 67e8b8be953a4a5784f2d098fd6364e3 (table default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb [id=7a2c513cd8e74817b4a391c9c7932085]). Expected version 1 got 0
I20240506 19:56:51.409972 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 768) took 2546 ms. Trace:
I20240506 19:56:51.410065 13246 rpcz_store.cc:276] 0506 19:56:48.863136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.863531 (+ 395us) service_pool.cc:225] Handling call
0506 19:56:51.276980 (+2413449us) ts_tablet_manager.cc:1120] Deleting tablet bafd3c4706f74e7da36a8f3538760f43
0506 19:56:51.276991 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.383189 (+106198us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.406265 (+ 23076us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.409947 (+ 3682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413245}
I20240506 19:56:51.410517 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 769) took 2547 ms. Trace:
I20240506 19:56:51.410225 12519 catalog_manager.cc:5484] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:51.410599 13246 rpcz_store.cc:276] 0506 19:56:48.863256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.863751 (+ 495us) service_pool.cc:225] Handling call
0506 19:56:51.410381 (+2546630us) ts_tablet_manager.cc:1120] Deleting tablet 4f382a89f94e4fdf9b8266d049635329
0506 19:56:51.410390 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.410499 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2546445}
I20240506 19:56:51.410764 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]). Expected version 3 got 0
I20240506 19:56:51.410851 13246 tablet_replica.cc:330] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.411044 13246 raft_consensus.cc:2229] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:51.411087 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]). Expected version 2 got 0
I20240506 19:56:51.411334 13246 pending_rounds.cc:62] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:51.411386 13246 pending_rounds.cc:68] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744078229671936 op_type: NO_OP noop_request { }
I20240506 19:56:51.411509 13246 raft_consensus.cc:2866] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:51.411597 13246 raft_consensus.cc:2258] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.412891 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet bafd3c4706f74e7da36a8f3538760f43 (table default.Table_162d0c1d88d04a11b0015afaea52e19f [id=5a2afe178f674b29834ccf87ec1ed218]) successfully deleted
W20240506 19:56:51.413177 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4f382a89f94e4fdf9b8266d049635329 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f382a89f94e4fdf9b8266d049635329 already in progress: creating tablet
I20240506 19:56:51.420369 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_a495d810a1ce4d3a80b146059120ad17 [id=16aa3036ed744751afd08515bb792ce2]). Expected version 1 got 0
I20240506 19:56:51.420712 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:51.420929 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:51.421088 14069 raft_consensus.cc:482] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.421252 14069 raft_consensus.cc:504] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.422071 14069 leader_election.cc:288] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.422307 12520 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 597c61f1d1104256b90f6601cf3e6667
I20240506 19:56:51.422637 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_28fc76ba360b4dd4b805761b13f00f04" } new_table_name: "default.Table_9c8b5a5db9104c929597de8778ef14a1" modify_external_catalogs: true
W20240506 19:56:51.423635 12783 leader_election.cc:341] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 88dba275893f4696a5318ec21f0b2477
W20240506 19:56:51.424000 12781 leader_election.cc:341] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 88dba275893f4696a5318ec21f0b2477
I20240506 19:56:51.424090 12781 leader_election.cc:302] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:51.424830 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 0d29c7c9c933414fb0ca6e6d370eac62 (DEFAULT_TABLE table=default.Table_1b1ba78c0a194d509cee193f3bec562e [id=af9eea916c5f44b2a0c650481873e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.425253 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d29c7c9c933414fb0ca6e6d370eac62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.426075 14069 raft_consensus.cc:2726] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.430305 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 5553e49640594e258d3506b9250a6c9f (DEFAULT_TABLE table=default.Table_4343df7e98714672b87cdc89128871a6 [id=5eae429c4d3c4a199359ea1c6532b6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.430881 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5553e49640594e258d3506b9250a6c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.431082 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:51.434229 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.434285 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.434229 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.436251 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c47cda70aad1490e8d9668e7240b0673"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.436961 13246 ts_tablet_manager.cc:1875] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.436923 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 8474271b3e334b50949d160676a61f0e (DEFAULT_TABLE table=default.Table_303d9190978b40a89152b877de75abb4 [id=7fe70815a63a40139593dcf8b27b60bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.437326 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8474271b3e334b50949d160676a61f0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.437546 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c47cda70aad1490e8d9668e7240b0673 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.438397 14223 raft_consensus.cc:482] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.438521 14223 raft_consensus.cc:504] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.439110 14223 leader_election.cc:288] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.439675 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5217c257ff46e48c23563e496b6f08" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:51.439803 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5217c257ff46e48c23563e496b6f08" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:51.440213 13057 leader_election.cc:341] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:51.440338 13058 leader_election.cc:341] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.440456 13058 leader_election.cc:302] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.440977 14223 raft_consensus.cc:2726] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.448488 13242 ts_tablet_manager.cc:1888] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.448735 13242 log.cc:1196] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/00de09ddcdd948a6a3500a6c9d814c14
I20240506 19:56:51.449175 13242 ts_tablet_manager.cc:1909] T 00de09ddcdd948a6a3500a6c9d814c14 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.449652 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 720) took 2780 ms. Trace:
I20240506 19:56:51.449846 13242 rpcz_store.cc:276] 0506 19:56:48.668859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.668918 (+ 59us) service_pool.cc:225] Handling call
0506 19:56:51.400219 (+2731301us) ts_tablet_manager.cc:1120] Deleting tablet 00de09ddcdd948a6a3500a6c9d814c14
0506 19:56:51.400232 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.435105 (+ 34873us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.448352 (+ 13247us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.449625 (+ 1273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2731092}
I20240506 19:56:51.450732 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 00de09ddcdd948a6a3500a6c9d814c14 (table default.Table_247909120dec47d3a2d1729c04ed16a6 [id=cc673f1dfa5d4469bb04e1d5a0b2d141]) successfully deleted
I20240506 19:56:51.451300 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 721) took 2720 ms. Trace:
I20240506 19:56:51.452108 13242 rpcz_store.cc:276] 0506 19:56:48.730508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.730875 (+ 367us) service_pool.cc:225] Handling call
0506 19:56:51.451131 (+2720256us) ts_tablet_manager.cc:1120] Deleting tablet 8689ec0a8e134bceb010053b1f5a343d
0506 19:56:51.451146 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.451275 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2720018}
W20240506 19:56:51.453145 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8689ec0a8e134bceb010053b1f5a343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8689ec0a8e134bceb010053b1f5a343d already in progress: creating tablet
I20240506 19:56:51.453718 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 722) took 2675 ms. Trace:
I20240506 19:56:51.457463 13242 rpcz_store.cc:276] 0506 19:56:48.777845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.778133 (+ 288us) service_pool.cc:225] Handling call
0506 19:56:51.453576 (+2675443us) ts_tablet_manager.cc:1120] Deleting tablet 9e17d0a261734d45866525b35de7cba3
0506 19:56:51.453589 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.453700 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2675247}
W20240506 19:56:51.458060 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9e17d0a261734d45866525b35de7cba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e17d0a261734d45866525b35de7cba3 already in progress: creating tablet
I20240506 19:56:51.459843 13242 tablet_replica.cc:330] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:51.460069 13242 raft_consensus.cc:2229] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.460278 13242 raft_consensus.cc:2258] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.463037 13242 ts_tablet_manager.cc:1875] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:51.464263 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ec1ac39f62b4e15a267f5db2a6166c6 consensus state has no leader
I20240506 19:56:51.464751 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ea1edc3350f3474c91f0828a99c2228f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.465448 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea1edc3350f3474c91f0828a99c2228f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.465386 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0551b9fce765497080cf8cad54f4dc9c" } new_table_name: "default.Table_e84e16df5a724b1e8fe6eb666bfa375a" modify_external_catalogs: true
W20240506 19:56:51.466868 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5553e49640594e258d3506b9250a6c9f consensus state has no leader
I20240506 19:56:51.467265 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_15973904471f431bb1fa23a0e060548a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.467787 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15973904471f431bb1fa23a0e060548a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.469331 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1403b3f7ba884b308e7346ed7eaa06c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.469820 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1403b3f7ba884b308e7346ed7eaa06c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.472442 14223 raft_consensus.cc:482] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.472597 14223 raft_consensus.cc:504] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.473358 14223 leader_election.cc:288] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.474066 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9296bb1104b4cbe279eecd3683796" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.474363 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9296bb1104b4cbe279eecd3683796" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:51.474745 13058 leader_election.cc:341] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:51.475075 13057 leader_election.cc:341] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.475180 13057 leader_election.cc:302] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.475442 14223 raft_consensus.cc:2726] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.479622 14069 raft_consensus.cc:482] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.479817 14069 raft_consensus.cc:504] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.480655 14069 leader_election.cc:288] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:51.482103 12781 leader_election.cc:341] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 104337875517445f862d82ece3a84889
W20240506 19:56:51.482338 12783 leader_election.cc:341] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 104337875517445f862d82ece3a84889
I20240506 19:56:51.482436 12783 leader_election.cc:302] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:51.483071 14069 raft_consensus.cc:2726] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.487596 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b6a622cd9e483ca707440b4759e48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.490702 13249 tablet_bootstrap.cc:492] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.490923 13249 ts_tablet_manager.cc:1367] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20240506 19:56:51.491993 13249 raft_consensus.cc:348] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.492280 13249 raft_consensus.cc:374] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.492364 13249 raft_consensus.cc:729] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.492774 13249 consensus_queue.cc:260] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.493371 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92329da0f9ec4d2e982efe5e9b94c5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.493518 13246 ts_tablet_manager.cc:1888] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:51.493722 13246 log.cc:1196] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/607b3fc2c17b4793af9b84c19066ec5e
I20240506 19:56:51.494894 13246 ts_tablet_manager.cc:1909] T 607b3fc2c17b4793af9b84c19066ec5e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.496989 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 778) took 2529 ms. Trace:
I20240506 19:56:51.497879 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_460654e013224b0b862d4c4663839cc1" } new_table_name: "default.Table_0bfd44a2bbd8485c82c64432fff1c61b" modify_external_catalogs: true
I20240506 19:56:51.498126 13246 rpcz_store.cc:276] 0506 19:56:48.967918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.967966 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:51.410808 (+2442842us) ts_tablet_manager.cc:1120] Deleting tablet 607b3fc2c17b4793af9b84c19066ec5e
0506 19:56:51.410816 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.447369 (+ 36553us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.491929 (+ 44560us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.496968 (+ 5039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442635}
I20240506 19:56:51.498567 13246 tablet_replica.cc:330] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.500093 13246 raft_consensus.cc:2229] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.501744 13246 raft_consensus.cc:2258] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.502261 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 607b3fc2c17b4793af9b84c19066ec5e (table default.Table_c3e795b5a9724754a33f79ebe9ac2c1f [id=343720fdaa3d43b28031433246aebe9c]) successfully deleted
I20240506 19:56:51.503906 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_81e0b435a0344362af8168947f3b1d56" } modify_external_catalogs: true
I20240506 19:56:51.504138 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_81e0b435a0344362af8168947f3b1d56" } modify_external_catalogs: true
I20240506 19:56:51.504756 13246 ts_tablet_manager.cc:1875] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.508208 13242 ts_tablet_manager.cc:1888] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.509320 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_07942e78a30c4f24b8a56a39c0f39e4c" } modify_external_catalogs: true
I20240506 19:56:51.509552 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_07942e78a30c4f24b8a56a39c0f39e4c" } modify_external_catalogs: true
I20240506 19:56:51.510046 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 841) took 1479 ms. Trace:
I20240506 19:56:51.510159 13243 rpcz_store.cc:276] 0506 19:56:50.030711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.048718 (+ 18007us) service_pool.cc:225] Handling call
0506 19:56:50.940830 (+892112us) ts_tablet_manager.cc:1120] Deleting tablet f8951bf4873e4382bcd42a52368fb376
0506 19:56:50.940841 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.036152 (+ 95311us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.383643 (+347491us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.510025 (+126382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":891917}
W20240506 19:56:51.510300 13057 consensus_peers.cc:485] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:51.510747 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 845) took 1398 ms. Trace:
I20240506 19:56:51.510829 13243 rpcz_store.cc:276] 0506 19:56:50.111830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.111908 (+ 78us) service_pool.cc:225] Handling call
0506 19:56:51.510625 (+1398717us) ts_tablet_manager.cc:1120] Deleting tablet f078559df48f4d23869ddcb4b3fca9fc
0506 19:56:51.510636 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.510730 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1398471}
I20240506 19:56:51.510964 14223 raft_consensus.cc:482] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.511226 13243 tablet_replica.cc:330] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:51.511085 14223 raft_consensus.cc:504] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.511413 13243 raft_consensus.cc:2229] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:51.511628 13243 pending_rounds.cc:62] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:51.511672 13243 pending_rounds.cc:68] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744082072465408 op_type: NO_OP noop_request { }
I20240506 19:56:51.511754 14223 leader_election.cc:288] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.511780 13243 raft_consensus.cc:2866] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:51.511858 13243 raft_consensus.cc:2258] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.512636 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3268094cbf481698cb373c1bf6dfba" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:51.513063 13057 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.513561 14223 raft_consensus.cc:482] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.513684 14223 raft_consensus.cc:504] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.514230 14223 leader_election.cc:288] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.514719 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:51.515157 13057 leader_election.cc:341] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.512370 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3268094cbf481698cb373c1bf6dfba" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.514781 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:51.517395 13058 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.517511 13058 leader_election.cc:302] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:51.517848 13058 leader_election.cc:341] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.517922 13058 leader_election.cc:302] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.518229 14211 raft_consensus.cc:2726] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.518592 14211 raft_consensus.cc:2726] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:51.518847 13242 log.cc:1196] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/20c1590e36b4471d8e72f4b07410ee0f
I20240506 19:56:51.519217 13242 ts_tablet_manager.cc:1909] T 20c1590e36b4471d8e72f4b07410ee0f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.522109 13245 tablet_bootstrap.cc:492] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.522259 13245 ts_tablet_manager.cc:1367] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.376s user 0.012s sys 0.009s
I20240506 19:56:51.522981 13245 raft_consensus.cc:348] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.523176 13245 raft_consensus.cc:374] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.523491 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f8951bf4873e4382bcd42a52368fb376 (table default.Table_6974ae6398ae4559a50fb236133424b1 [id=07a119f6e5854b1090f2f47dc8c9dada]) successfully deleted
W20240506 19:56:51.523769 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f078559df48f4d23869ddcb4b3fca9fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f078559df48f4d23869ddcb4b3fca9fc already in progress: creating tablet
I20240506 19:56:51.523229 13245 raft_consensus.cc:729] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.526026 13245 consensus_queue.cc:260] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.526706 13245 ts_tablet_manager.cc:1398] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:51.529486 13245 tablet_bootstrap.cc:492] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.530020 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 723) took 2752 ms. Trace:
I20240506 19:56:51.530198 13242 rpcz_store.cc:276] 0506 19:56:48.777963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.778357 (+ 394us) service_pool.cc:225] Handling call
0506 19:56:51.459786 (+2681429us) ts_tablet_manager.cc:1120] Deleting tablet 20c1590e36b4471d8e72f4b07410ee0f
0506 19:56:51.459798 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.490201 (+ 30403us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.508088 (+ 17887us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.529991 (+ 21903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2681233}
I20240506 19:56:51.531174 13245 tablet_bootstrap.cc:654] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.531625 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 20c1590e36b4471d8e72f4b07410ee0f (table default.Table_1a0e2e4b842f4375ab63b84f691c796b [id=d913e8d66e1b47f7a69d9a48d03c1722]) successfully deleted
I20240506 19:56:51.532178 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 724) took 2754 ms. Trace:
I20240506 19:56:51.532327 13242 rpcz_store.cc:276] 0506 19:56:48.778078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.778572 (+ 494us) service_pool.cc:225] Handling call
0506 19:56:51.532019 (+2753447us) ts_tablet_manager.cc:1120] Deleting tablet c5f3def738fb4cb195d08e48d7d2b293
0506 19:56:51.532034 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.532156 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753284}
W20240506 19:56:51.534492 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c5f3def738fb4cb195d08e48d7d2b293 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5f3def738fb4cb195d08e48d7d2b293 already in progress: creating tablet
I20240506 19:56:51.535055 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 725) took 2741 ms. Trace:
I20240506 19:56:51.538551 13242 rpcz_store.cc:276] 0506 19:56:48.793152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.793467 (+ 315us) service_pool.cc:225] Handling call
0506 19:56:51.534903 (+2741436us) ts_tablet_manager.cc:1120] Deleting tablet 8689ec0a8e134bceb010053b1f5a343d
0506 19:56:51.534917 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.535034 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2734126}
W20240506 19:56:51.539270 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8689ec0a8e134bceb010053b1f5a343d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8689ec0a8e134bceb010053b1f5a343d already in progress: creating tablet
I20240506 19:56:51.539690 13249 ts_tablet_manager.cc:1398] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:51.535741 13244 tablet_bootstrap.cc:492] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:51.540022 13244 ts_tablet_manager.cc:1367] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.496s user 0.012s sys 0.026s
I20240506 19:56:51.540208 13249 tablet_bootstrap.cc:492] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.541119 13244 raft_consensus.cc:348] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.541738 13244 raft_consensus.cc:374] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.541815 13244 raft_consensus.cc:729] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.542088 13244 consensus_queue.cc:260] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.544302 13249 tablet_bootstrap.cc:654] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.551659 13243 ts_tablet_manager.cc:1875] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.555002 13244 ts_tablet_manager.cc:1398] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240506 19:56:51.561877 13244 tablet_bootstrap.cc:492] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:51.563916 13244 tablet_bootstrap.cc:654] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.566746 14250 consensus_queue.cc:1035] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:51.569171 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 729) took 2758 ms. Trace:
W20240506 19:56:51.569612 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93d6e4b9f0314cf19bbdd9fd9038d97d consensus state has no leader
I20240506 19:56:51.569272 13242 rpcz_store.cc:276] 0506 19:56:48.810276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.810330 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:51.569028 (+2758698us) ts_tablet_manager.cc:1120] Deleting tablet e1d7a850a3e644c3a38ab224efcc8af5
0506 19:56:51.569040 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.569148 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2758499}
I20240506 19:56:51.570428 14250 raft_consensus.cc:482] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.570564 14250 raft_consensus.cc:504] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:51.570629 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet e1d7a850a3e644c3a38ab224efcc8af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d7a850a3e644c3a38ab224efcc8af5 already in progress: creating tablet
I20240506 19:56:51.571566 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acde41b2c03440a89243aae008779e1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.572222 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acde41b2c03440a89243aae008779e1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:51.572306 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 728) took 2762 ms. Trace:
I20240506 19:56:51.572443 13120 raft_consensus.cc:2454] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:51.572404 13242 rpcz_store.cc:276] 0506 19:56:48.810155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.810558 (+ 403us) service_pool.cc:225] Handling call
0506 19:56:51.572189 (+2761631us) ts_tablet_manager.cc:1120] Deleting tablet ce6483470c864697be6d9590ec95edfc
0506 19:56:51.572198 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.572290 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2761475}
W20240506 19:56:51.572839 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ce6483470c864697be6d9590ec95edfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce6483470c864697be6d9590ec95edfc already in progress: creating tablet
W20240506 19:56:51.572916 12921 leader_election.cc:341] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.573266 12922 leader_election.cc:302] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.573581 14080 raft_consensus.cc:2781] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.573657 14080 raft_consensus.cc:482] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.573716 14080 raft_consensus.cc:3037] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.576325 14080 raft_consensus.cc:504] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.576740 13246 ts_tablet_manager.cc:1888] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.576869 13246 log.cc:1196] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:51.577101 12521 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]). Expected version 1 got 0
I20240506 19:56:51.577204 13246 ts_tablet_manager.cc:1909] T 2f7f7171d06948388bc0fdb3514202ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.577750 14080 leader_election.cc:288] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.577713 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acde41b2c03440a89243aae008779e1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:51.578415 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 730) took 2767 ms. Trace:
I20240506 19:56:51.578460 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acde41b2c03440a89243aae008779e1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:51.578624 13120 raft_consensus.cc:3037] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.578526 13242 rpcz_store.cc:276] 0506 19:56:48.811200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.811412 (+ 212us) service_pool.cc:225] Handling call
0506 19:56:51.578264 (+2766852us) ts_tablet_manager.cc:1120] Deleting tablet a8448ad8e9384edba7202d0bdce516a4
0506 19:56:51.578277 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.578395 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2766675}
W20240506 19:56:51.579113 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a8448ad8e9384edba7202d0bdce516a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8448ad8e9384edba7202d0bdce516a4 already in progress: creating tablet
I20240506 19:56:51.580159 13242 tablet_replica.cc:330] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:51.580955 13242 raft_consensus.cc:2229] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.581326 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet da41ef03182445f1946584c2bf015139 with delete_type TABLET_DATA_DELETED (replaced by tablet 472c63d8995346d78633154a8f0d4c2b) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.581808 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_195d90ff34ff4ca6b6282483c9691f47" } modify_external_catalogs: true
W20240506 19:56:51.582129 12921 leader_election.cc:341] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.582020 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_195d90ff34ff4ca6b6282483c9691f47" } modify_external_catalogs: true
I20240506 19:56:51.582376 13120 raft_consensus.cc:2454] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:51.582862 12922 leader_election.cc:302] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.583282 14080 raft_consensus.cc:2781] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.583575 14080 raft_consensus.cc:686] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:51.583847 14080 consensus_queue.cc:237] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.584897 14250 leader_election.cc:288] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.585084 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f04d735a43fb445c9e751f02de612410" } modify_external_catalogs: true
I20240506 19:56:51.585279 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f04d735a43fb445c9e751f02de612410" } modify_external_catalogs: true
I20240506 19:56:51.588586 13245 tablet_bootstrap.cc:492] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.588771 13245 ts_tablet_manager.cc:1367] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.002s
I20240506 19:56:51.589777 13245 raft_consensus.cc:348] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.589954 13245 raft_consensus.cc:374] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.589996 13245 raft_consensus.cc:729] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.590240 13245 consensus_queue.cc:260] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.591421 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 20c1590e36b4471d8e72f4b07410ee0f from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:51.591885 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 15ed0c7b4569490b91c54d5bdb187e49 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:51.592481 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:51.592736 13242 raft_consensus.cc:2258] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.595188 13245 ts_tablet_manager.cc:1398] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:51.596021 13242 ts_tablet_manager.cc:1875] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:51.596940 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 418e02201bfc4a34a22612c2edc3e05f consensus state has no leader
W20240506 19:56:51.598389 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 196b339d0bc54426b69a60fe59f1af00 consensus state has no leader
I20240506 19:56:51.599464 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d43da1d12ce647dcbcd31263932b2f54
I20240506 19:56:51.600638 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet d43da1d12ce647dcbcd31263932b2f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.601783 13245 tablet_bootstrap.cc:492] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.603391 13245 tablet_bootstrap.cc:654] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.603451 12515 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]). Expected version 1 got 0
I20240506 19:56:51.605473 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet d43da1d12ce647dcbcd31263932b2f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.605777 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 783) took 2547 ms. Trace:
I20240506 19:56:51.605894 13246 rpcz_store.cc:276] 0506 19:56:49.058041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.058101 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:51.498514 (+2440413us) ts_tablet_manager.cc:1120] Deleting tablet 2f7f7171d06948388bc0fdb3514202ed
0506 19:56:51.498527 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.531034 (+ 32507us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.576612 (+ 45578us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.605750 (+ 29138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2440207}
I20240506 19:56:51.606540 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2f7f7171d06948388bc0fdb3514202ed (table default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]) successfully deleted
I20240506 19:56:51.609367 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 691e312510a440898a58bec43ab26e8f
I20240506 19:56:51.610527 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e312510a440898a58bec43ab26e8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.611667 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e312510a440898a58bec43ab26e8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.613652 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet d43da1d12ce647dcbcd31263932b2f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:51.614073 12921 consensus_peers.cc:485] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:51.614249 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e312510a440898a58bec43ab26e8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.614655 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_99bde890b8c14fe2b57afaa1b6f6ed45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.615382 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99bde890b8c14fe2b57afaa1b6f6ed45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.628299 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 6a7715dad8184493b085d891428ad44c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:51.628427 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 785) took 2536 ms. Trace:
I20240506 19:56:51.628522 13246 rpcz_store.cc:276] 0506 19:56:49.092033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.092084 (+ 51us) service_pool.cc:225] Handling call
0506 19:56:51.606270 (+2514186us) ts_tablet_manager.cc:1120] Deleting tablet d6edff8e7e6945aa926aa1486cbbc2bc
0506 19:56:51.628224 (+ 21954us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.628398 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2513994}
W20240506 19:56:51.629093 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d6edff8e7e6945aa926aa1486cbbc2bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6edff8e7e6945aa926aa1486cbbc2bc already in progress: creating tablet
I20240506 19:56:51.631361 13244 tablet_bootstrap.cc:492] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:51.631644 13244 ts_tablet_manager.cc:1367] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.002s
I20240506 19:56:51.632333 13244 raft_consensus.cc:348] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.632539 13244 raft_consensus.cc:374] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.632584 13244 raft_consensus.cc:729] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.633180 13244 consensus_queue.cc:260] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.641563 12516 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:51.642633 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_964a0d50cb2d48fa988072257bec9ec2" } modify_external_catalogs: true
I20240506 19:56:51.642836 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_964a0d50cb2d48fa988072257bec9ec2" } modify_external_catalogs: true
I20240506 19:56:51.649756 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet af2741336d5a4be1b5306319fc20cdc6
I20240506 19:56:51.651137 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet af2741336d5a4be1b5306319fc20cdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.652860 13244 ts_tablet_manager.cc:1398] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240506 19:56:51.657187 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet af2741336d5a4be1b5306319fc20cdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.657996 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e6d37bb1c2144c4fa71aab6113b9f06e
I20240506 19:56:51.658193 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet af2741336d5a4be1b5306319fc20cdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.658679 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d37bb1c2144c4fa71aab6113b9f06e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.659257 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d37bb1c2144c4fa71aab6113b9f06e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.659608 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d37bb1c2144c4fa71aab6113b9f06e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.662431 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 784) took 2570 ms. Trace:
I20240506 19:56:51.662572 13246 rpcz_store.cc:276] 0506 19:56:49.091888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.092302 (+ 414us) service_pool.cc:225] Handling call
0506 19:56:51.628902 (+2536600us) ts_tablet_manager.cc:1120] Deleting tablet 08535fa1600b41ff919fefa93cfc4c78
0506 19:56:51.662261 (+ 33359us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.662410 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2536477}
W20240506 19:56:51.663111 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 08535fa1600b41ff919fefa93cfc4c78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08535fa1600b41ff919fefa93cfc4c78 already in progress: creating tablet
I20240506 19:56:51.665994 13245 tablet_bootstrap.cc:492] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:51.667106 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 58109.9us
I20240506 19:56:51.667094 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_08345b91e36e40d7860b550554026c61" } new_table_name: "default.Table_ece1e5630f42487eb58e4d048c7bbc54" modify_external_catalogs: true
I20240506 19:56:51.667012 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6cc42687d6c545068a39c096f2932dcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.667771 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cc42687d6c545068a39c096f2932dcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.670500 13244 tablet_bootstrap.cc:492] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:51.672770 13245 ts_tablet_manager.cc:1367] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.002s
I20240506 19:56:51.673602 13245 raft_consensus.cc:348] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.673856 13245 raft_consensus.cc:374] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.673966 13245 raft_consensus.cc:729] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.677122 13244 tablet_bootstrap.cc:654] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.678725 13245 consensus_queue.cc:260] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.679389 13245 ts_tablet_manager.cc:1398] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:51.680493 13242 ts_tablet_manager.cc:1888] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.682691 13242 log.cc:1196] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/cc8335f9308046279a25a4a95d8e0f65
I20240506 19:56:51.683034 13242 ts_tablet_manager.cc:1909] T cc8335f9308046279a25a4a95d8e0f65 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.683377 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 731) took 2861 ms. Trace:
I20240506 19:56:51.683459 13242 rpcz_store.cc:276] 0506 19:56:48.822075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.834252 (+ 12177us) service_pool.cc:225] Handling call
0506 19:56:51.580099 (+2745847us) ts_tablet_manager.cc:1120] Deleting tablet cc8335f9308046279a25a4a95d8e0f65
0506 19:56:51.580112 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.672386 (+ 92274us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.680374 (+ 7988us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.683352 (+ 2978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745647}
I20240506 19:56:51.683867 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 736) took 2820 ms. Trace:
I20240506 19:56:51.683940 13242 rpcz_store.cc:276] 0506 19:56:48.863435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:48.863501 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:51.683739 (+2820238us) ts_tablet_manager.cc:1120] Deleting tablet 4f382a89f94e4fdf9b8266d049635329
0506 19:56:51.683749 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.683850 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2820063}
I20240506 19:56:51.684319 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 749) took 2602 ms. Trace:
I20240506 19:56:51.684405 13242 rpcz_store.cc:276] 0506 19:56:49.082271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.082324 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:51.684194 (+2601870us) ts_tablet_manager.cc:1120] Deleting tablet 08535fa1600b41ff919fefa93cfc4c78
0506 19:56:51.684204 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.684301 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2601672}
I20240506 19:56:51.684595 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c8eeaa71ac61424796c6ffdd993de9db
I20240506 19:56:51.684702 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 750) took 2600 ms. Trace:
I20240506 19:56:51.684778 13242 rpcz_store.cc:276] 0506 19:56:49.084358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.084411 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:51.684612 (+2600201us) ts_tablet_manager.cc:1120] Deleting tablet d6edff8e7e6945aa926aa1486cbbc2bc
0506 19:56:51.684620 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.684689 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2600000}
I20240506 19:56:51.684608 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c1a5a8dc9a2a4856b8b8f2aaac187c36
I20240506 19:56:51.685016 13242 tablet_replica.cc:330] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:51.685192 13242 raft_consensus.cc:2229] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.685782 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 with delete_type TABLET_DATA_DELETED (replaced by tablet b904072d89c6486fa3ca1557220f0b52) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.688966 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet c8eeaa71ac61424796c6ffdd993de9db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.686434 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet c8eeaa71ac61424796c6ffdd993de9db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.689456 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 with delete_type TABLET_DATA_DELETED (replaced by tablet b904072d89c6486fa3ca1557220f0b52) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.687546 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_07709e08656c44c892d8a4f181d10bd0" } new_table_name: "default.Table_f13896e2670e40119184cffaa84b47d9" modify_external_catalogs: true
I20240506 19:56:51.684010 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet cc8335f9308046279a25a4a95d8e0f65 (table default.Table_df1733520c4a4ae6a99a78f194d6cf06 [id=64a093df5c7143668da7d3e91eccba2a]) successfully deleted
W20240506 19:56:51.691726 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4f382a89f94e4fdf9b8266d049635329 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f382a89f94e4fdf9b8266d049635329 already in progress: creating tablet
W20240506 19:56:51.691958 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 08535fa1600b41ff919fefa93cfc4c78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08535fa1600b41ff919fefa93cfc4c78 already in progress: creating tablet
W20240506 19:56:51.692183 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d6edff8e7e6945aa926aa1486cbbc2bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6edff8e7e6945aa926aa1486cbbc2bc already in progress: creating tablet
I20240506 19:56:51.688060 12522 catalog_manager.cc:6554] ReplaceTablet: tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 deleted and replaced by tablet b904072d89c6486fa3ca1557220f0b52
I20240506 19:56:51.688673 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6f73fec327d24cf7bfabb15d38171100"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.693079 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f73fec327d24cf7bfabb15d38171100 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.693733 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet c8eeaa71ac61424796c6ffdd993de9db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.693744 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 with delete_type TABLET_DATA_DELETED (replaced by tablet b904072d89c6486fa3ca1557220f0b52) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.695529 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 786) took 2601 ms. Trace:
I20240506 19:56:51.695669 13246 rpcz_store.cc:276] 0506 19:56:49.093706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.094172 (+ 466us) service_pool.cc:225] Handling call
0506 19:56:51.663471 (+2569299us) ts_tablet_manager.cc:1120] Deleting tablet fdef0a40f2e346a798b01e83a960de09
0506 19:56:51.695369 (+ 31898us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.695503 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2569126}
I20240506 19:56:51.689621 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:51.696051 13246 tablet_replica.cc:330] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:51.698566 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5751124adb5b40f59e8d79a4605ae8c4 consensus state has no leader
I20240506 19:56:51.696280 13246 raft_consensus.cc:2229] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.699354 13246 raft_consensus.cc:2258] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:51.699978 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet fdef0a40f2e346a798b01e83a960de09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdef0a40f2e346a798b01e83a960de09 already in progress: creating tablet
I20240506 19:56:51.700127 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3a9d0dcf4191468da0f575d88f4a47f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.700773 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a9d0dcf4191468da0f575d88f4a47f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.701589 13246 ts_tablet_manager.cc:1875] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.704586 13245 tablet_bootstrap.cc:492] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:51.705354 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ec5458abb234366af4c23754a54d68a consensus state has no leader
I20240506 19:56:51.689623 13244 tablet_bootstrap.cc:492] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:51.707679 13244 ts_tablet_manager.cc:1367] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240506 19:56:51.707721 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c5bb44d113914983a38052bc548f4eac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.708465 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5bb44d113914983a38052bc548f4eac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.710283 13244 raft_consensus.cc:348] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.713989 13244 raft_consensus.cc:374] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.714111 13244 raft_consensus.cc:729] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.714344 13245 tablet_bootstrap.cc:654] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.714897 13244 consensus_queue.cc:260] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.720645 13243 ts_tablet_manager.cc:1888] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:51.726647 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f3e7df09f347bfa5cad4de34dbda13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.730137 12520 catalog_manager.cc:5484] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:51.730701 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 67e8b8be953a4a5784f2d098fd6364e3 (table default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb [id=7a2c513cd8e74817b4a391c9c7932085]). Expected version 1 got 0
I20240506 19:56:51.731045 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]). Expected version 3 got 0
I20240506 19:56:51.731298 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet d3ec577dd06640068575288c24e1105a (table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]). Expected version 1 got 0
I20240506 19:56:51.731866 12520 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:51.732656 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 6d815013b9f54acb8a978137961921a1 (DEFAULT_TABLE table=default.Table_2a2d910a263840e7934d2972c9cd8a87 [id=0f8f012e7cf24efe9c4a08abf3fab067]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.732863 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]). Expected version 2 got 0
I20240506 19:56:51.733153 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d815013b9f54acb8a978137961921a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.737872 13244 ts_tablet_manager.cc:1398] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240506 19:56:51.741909 13244 tablet_bootstrap.cc:492] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:51.748888 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 6d815013b9f54acb8a978137961921a1 (DEFAULT_TABLE table=default.Table_2a2d910a263840e7934d2972c9cd8a87 [id=0f8f012e7cf24efe9c4a08abf3fab067]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.750826 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d815013b9f54acb8a978137961921a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.754628 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 15ed0c7b4569490b91c54d5bdb187e49
I20240506 19:56:51.755239 12513 catalog_manager.cc:6554] ReplaceTablet: tablet 15ed0c7b4569490b91c54d5bdb187e49 deleted and replaced by tablet abdd4edb2c454681a65e363c88941f58
I20240506 19:56:51.755528 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ed0c7b4569490b91c54d5bdb187e49 with delete_type TABLET_DATA_DELETED (replaced by tablet abdd4edb2c454681a65e363c88941f58) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.756050 13242 raft_consensus.cc:2258] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.756294 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ed0c7b4569490b91c54d5bdb187e49 with delete_type TABLET_DATA_DELETED (replaced by tablet abdd4edb2c454681a65e363c88941f58) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.756645 13242 ts_tablet_manager.cc:1875] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.756897 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:51.757752 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 60f07ed7653f4910b3a47890842216bc (DEFAULT_TABLE table=default.Table_10658c410e1648df879bdf21d2293b2e [id=b4856100088a41d083991b82b8185f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.759593 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 60f07ed7653f4910b3a47890842216bc (DEFAULT_TABLE table=default.Table_10658c410e1648df879bdf21d2293b2e [id=b4856100088a41d083991b82b8185f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.760052 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f07ed7653f4910b3a47890842216bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.762964 13244 tablet_bootstrap.cc:654] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.764597 13246 ts_tablet_manager.cc:1888] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:51.764746 13246 log.cc:1196] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/63f56d2180b1472a8168fb3a63589065
I20240506 19:56:51.765125 13246 ts_tablet_manager.cc:1909] T 63f56d2180b1472a8168fb3a63589065 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:51.765980 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 787) took 2639 ms. Trace:
I20240506 19:56:51.767782 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 6a7715dad8184493b085d891428ad44c
I20240506 19:56:51.768252 12515 catalog_manager.cc:6554] ReplaceTablet: tablet 6a7715dad8184493b085d891428ad44c deleted and replaced by tablet 912fa5b484e74572861f58ce78b1572e
I20240506 19:56:51.769286 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7715dad8184493b085d891428ad44c with delete_type TABLET_DATA_DELETED (replaced by tablet 912fa5b484e74572861f58ce78b1572e) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.766100 13246 rpcz_store.cc:276] 0506 19:56:49.126429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.126625 (+ 196us) service_pool.cc:225] Handling call
0506 19:56:51.695994 (+2569369us) ts_tablet_manager.cc:1120] Deleting tablet 63f56d2180b1472a8168fb3a63589065
0506 19:56:51.696003 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.713543 (+ 17540us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.764483 (+ 50940us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.765911 (+ 1428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2569157}
I20240506 19:56:51.770056 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 788) took 2643 ms. Trace:
I20240506 19:56:51.770133 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 63f56d2180b1472a8168fb3a63589065 (table default.Table_ab69ac48aa7a4934a9f383276277cf10 [id=493f88f528114491b898c4a1156712fc]) successfully deleted
I20240506 19:56:51.770195 13246 rpcz_store.cc:276] 0506 19:56:49.126626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.126853 (+ 227us) service_pool.cc:225] Handling call
0506 19:56:51.769863 (+2643010us) ts_tablet_manager.cc:1120] Deleting tablet 79592a4ecacb4324b2278e1b9a75496c
0506 19:56:51.769874 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.769986 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2642859}
I20240506 19:56:51.770545 13246 tablet_replica.cc:330] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:51.770785 13246 raft_consensus.cc:2229] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:51.770799 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 79592a4ecacb4324b2278e1b9a75496c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79592a4ecacb4324b2278e1b9a75496c already in progress: creating tablet
I20240506 19:56:51.770993 13246 raft_consensus.cc:2258] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.770792 13245 tablet_bootstrap.cc:492] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.771565 13245 ts_tablet_manager.cc:1367] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.001s
I20240506 19:56:51.772646 13246 ts_tablet_manager.cc:1875] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.723022 13249 tablet_bootstrap.cc:492] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.735713 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 6d815013b9f54acb8a978137961921a1 (DEFAULT_TABLE table=default.Table_2a2d910a263840e7934d2972c9cd8a87 [id=0f8f012e7cf24efe9c4a08abf3fab067]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.774397 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d815013b9f54acb8a978137961921a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.774345 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a495d810a1ce4d3a80b146059120ad17" } new_table_name: "default.Table_d529a0b0f9774e0385d3b2319db222dc" modify_external_catalogs: true
I20240506 19:56:51.774041 13243 log.cc:1196] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:51.775935 13243 ts_tablet_manager.cc:1909] T 2f7f7171d06948388bc0fdb3514202ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:51.756013 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ed0c7b4569490b91c54d5bdb187e49 with delete_type TABLET_DATA_DELETED (replaced by tablet abdd4edb2c454681a65e363c88941f58) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.772414 13245 raft_consensus.cc:348] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.778169 13245 raft_consensus.cc:374] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.778301 13245 raft_consensus.cc:729] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.778271 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_32db184637f04319ae8841be3169f1e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.778589 13245 consensus_queue.cc:260] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.756289 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 60f07ed7653f4910b3a47890842216bc (DEFAULT_TABLE table=default.Table_10658c410e1648df879bdf21d2293b2e [id=b4856100088a41d083991b82b8185f1c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:51.778873 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32db184637f04319ae8841be3169f1e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.779158 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f07ed7653f4910b3a47890842216bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.779289 13245 ts_tablet_manager.cc:1398] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:56:51.779579 13245 tablet_bootstrap.cc:492] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.769549 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 3141b13136c147b8b9d199c9728208e5 (DEFAULT_TABLE table=default.Table_9caff027d79b4a8cbdde4ba6074c1e90 [id=3f77354fa69f4941abc4db4fd5b70b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.769783 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7715dad8184493b085d891428ad44c with delete_type TABLET_DATA_DELETED (replaced by tablet 912fa5b484e74572861f58ce78b1572e) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.771801 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 with delete_type TABLET_DATA_DELETED (replaced by tablet 986f9c83b04f41b49e9b78fce731a90f) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.772738 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acde41b2c03440a89243aae008779e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.781591 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3141b13136c147b8b9d199c9728208e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.781585 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7715dad8184493b085d891428ad44c with delete_type TABLET_DATA_DELETED (replaced by tablet 912fa5b484e74572861f58ce78b1572e) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.781855 13249 ts_tablet_manager.cc:1367] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.242s user 0.004s sys 0.000s
I20240506 19:56:51.781945 13245 tablet_bootstrap.cc:654] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.782649 13249 raft_consensus.cc:348] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.783037 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 3141b13136c147b8b9d199c9728208e5 (DEFAULT_TABLE table=default.Table_9caff027d79b4a8cbdde4ba6074c1e90 [id=3f77354fa69f4941abc4db4fd5b70b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.784744 13249 raft_consensus.cc:374] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.784868 13249 raft_consensus.cc:729] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.785169 13249 consensus_queue.cc:260] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.788668 13249 ts_tablet_manager.cc:1398] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240506 19:56:51.789026 13249 tablet_bootstrap.cc:492] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.785418 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 3141b13136c147b8b9d199c9728208e5 (DEFAULT_TABLE table=default.Table_9caff027d79b4a8cbdde4ba6074c1e90 [id=3f77354fa69f4941abc4db4fd5b70b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.790544 13249 tablet_bootstrap.cc:654] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.793387 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:51.794643 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cd4a27060d284b228e7949d9118b32ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.795281 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd4a27060d284b228e7949d9118b32ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.799278 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3141b13136c147b8b9d199c9728208e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.805189 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fdebb50309c74888b6a893ac37f9cb9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.806149 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdebb50309c74888b6a893ac37f9cb9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.807428 13245 tablet_bootstrap.cc:492] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.807632 13245 ts_tablet_manager.cc:1367] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20240506 19:56:51.808108 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 0bcbf850d4914899a72f3b786d8b80b0 (DEFAULT_TABLE table=default.Table_78690cea0de8499bb46d757158d49954 [id=cc964ae4c3df4c588af2815988deb7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.808535 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bcbf850d4914899a72f3b786d8b80b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.808496 13245 raft_consensus.cc:348] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.808727 13245 raft_consensus.cc:374] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.808812 13245 raft_consensus.cc:729] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.808961 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cd0e12d470dd429295bcb3d354afacfe" } modify_external_catalogs: true
I20240506 19:56:51.809056 13245 consensus_queue.cc:260] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.809557 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2f9d2a9b611448949d0a7fc4f5146078" } new_table_name: "default.Table_1bb7f42c11b54b52b98a8181cdb076d3" modify_external_catalogs: true
I20240506 19:56:51.809748 13245 ts_tablet_manager.cc:1398] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240506 19:56:51.810062 13245 tablet_bootstrap.cc:492] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.810142 12513 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:51.809178 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cd0e12d470dd429295bcb3d354afacfe" } modify_external_catalogs: true
I20240506 19:56:51.811532 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_eb4ba3a6295e43989a12e36aa7dcc41a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.812050 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb4ba3a6295e43989a12e36aa7dcc41a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.812247 13245 tablet_bootstrap.cc:654] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.812672 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d59316fe0b304bd9b03da5f95408a1c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.815477 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d59316fe0b304bd9b03da5f95408a1c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.815701 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc with delete_type TABLET_DATA_DELETED (replaced by tablet efc41afd84a74652a82cc639bf562d99) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.817164 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb" } modify_external_catalogs: true
I20240506 19:56:51.817382 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb" } modify_external_catalogs: true
I20240506 19:56:51.824254 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fadc9a32c1fb42dc858cd67f8e78e34f" } new_table_name: "default.Table_e300dc01d2fc4eb1bb7c22f60f49f570" modify_external_catalogs: true
I20240506 19:56:51.826381 13245 tablet_bootstrap.cc:492] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.826560 13245 ts_tablet_manager.cc:1367] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20240506 19:56:51.827409 13245 raft_consensus.cc:348] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.827627 13245 raft_consensus.cc:374] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.827731 13245 raft_consensus.cc:729] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.827961 13245 consensus_queue.cc:260] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:51.828666 13245 ts_tablet_manager.cc:1398] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:51.828953 13245 tablet_bootstrap.cc:492] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:51.829809 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 consensus state has no leader
I20240506 19:56:51.830756 13245 tablet_bootstrap.cc:654] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.832216 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 0bcbf850d4914899a72f3b786d8b80b0 (DEFAULT_TABLE table=default.Table_78690cea0de8499bb46d757158d49954 [id=cc964ae4c3df4c588af2815988deb7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.832731 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.836727 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 0bcbf850d4914899a72f3b786d8b80b0 (DEFAULT_TABLE table=default.Table_78690cea0de8499bb46d757158d49954 [id=cc964ae4c3df4c588af2815988deb7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.838253 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bcbf850d4914899a72f3b786d8b80b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.840490 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 846) took 1598 ms. Trace:
I20240506 19:56:51.840652 13243 rpcz_store.cc:276] 0506 19:56:50.241817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.249440 (+ 7623us) service_pool.cc:225] Handling call
0506 19:56:51.511180 (+1261740us) ts_tablet_manager.cc:1120] Deleting tablet 2f7f7171d06948388bc0fdb3514202ed
0506 19:56:51.511188 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.551629 (+ 40441us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x616000025054. stack: 00007f2640ce3756 00007f2640ce3510 00007f263fd1648d 0000000000955804 00000000009557cd 000000000095519a 00007f264c5591aa 00007f26502bf5be 00007f26502f1844 00007f26502d2b30 00007f26502d28bd 0000000000a60bcc 00007f2640d29b74 00007f2640d2d19b 00007f2640d2d018 0000000000a60bcc
0506 19:56:51.637132 (+ 85503us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.720525 (+ 83393us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.840454 (+119929us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37461376,"tablet-delete.queue_time_us":1213267}
I20240506 19:56:51.841408 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2f7f7171d06948388bc0fdb3514202ed (table default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]) successfully deleted
I20240506 19:56:51.841922 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 864) took 1236 ms. Trace:
I20240506 19:56:51.842065 13243 rpcz_store.cc:276] 0506 19:56:50.605600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.605689 (+ 89us) service_pool.cc:225] Handling call
0506 19:56:51.841764 (+1236075us) ts_tablet_manager.cc:1120] Deleting tablet 110a046c2db14758b8d28616ae63c18c
0506 19:56:51.841777 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.841904 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235860}
W20240506 19:56:51.842644 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 110a046c2db14758b8d28616ae63c18c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a046c2db14758b8d28616ae63c18c already in progress: creating tablet
I20240506 19:56:51.843060 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 865) took 1237 ms. Trace:
I20240506 19:56:51.843179 13243 rpcz_store.cc:276] 0506 19:56:50.605726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.605819 (+ 93us) service_pool.cc:225] Handling call
0506 19:56:51.842954 (+1237135us) ts_tablet_manager.cc:1120] Deleting tablet 269210fbc3d4446181b123f200aae472
0506 19:56:51.842965 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.843043 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236966}
I20240506 19:56:51.843397 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a94ecb64b4b64889b3585e64fff5a (DEFAULT_TABLE table=default.Table_110baf1a2d08406abaddd6e76dd187a4 [id=7ce9557d82c14955b66d3970197b5bee]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:51.843653 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 269210fbc3d4446181b123f200aae472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 269210fbc3d4446181b123f200aae472 already in progress: creating tablet
I20240506 19:56:51.844081 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 866) took 1237 ms. Trace:
I20240506 19:56:51.844197 13243 rpcz_store.cc:276] 0506 19:56:50.606168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.606213 (+ 45us) service_pool.cc:225] Handling call
0506 19:56:51.843972 (+1237759us) ts_tablet_manager.cc:1120] Deleting tablet 0bc56299ed854d6c8ef201127692fd60
0506 19:56:51.843981 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.844066 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237592}
W20240506 19:56:51.844673 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0bc56299ed854d6c8ef201127692fd60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bc56299ed854d6c8ef201127692fd60 already in progress: creating tablet
I20240506 19:56:51.849611 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a94ecb64b4b64889b3585e64fff5a (DEFAULT_TABLE table=default.Table_110baf1a2d08406abaddd6e76dd187a4 [id=7ce9557d82c14955b66d3970197b5bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.850093 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a94ecb64b4b64889b3585e64fff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.850811 14223 raft_consensus.cc:482] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.851011 14223 raft_consensus.cc:504] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.851701 14223 leader_election.cc:288] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.852416 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c7c7b2da9e403f8241a512e210c4d9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:51.852420 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c7c7b2da9e403f8241a512e210c4d9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.852591 12982 raft_consensus.cc:2454] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:51.853000 13058 leader_election.cc:341] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.853405 13057 leader_election.cc:302] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.853518 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 57a58ed1c4bd4f80834f9c6f52355334 (DEFAULT_TABLE table=default.Table_4450908020fd475c8c535415af0a8925 [id=5c6305fd4cde4b1a87ae85c9267d759e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.853866 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a58ed1c4bd4f80834f9c6f52355334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.854136 14223 raft_consensus.cc:2781] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.854234 14223 raft_consensus.cc:482] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.854344 14223 raft_consensus.cc:3037] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.855432 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 57a58ed1c4bd4f80834f9c6f52355334 (DEFAULT_TABLE table=default.Table_4450908020fd475c8c535415af0a8925 [id=5c6305fd4cde4b1a87ae85c9267d759e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.860983 13123 raft_consensus.cc:1261] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 19:56:51.863948 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 44ce8749846847108edf5f554139cbd9
W20240506 19:56:51.864013 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b803ae289344f88b604a2053881d59b consensus state has no leader
W20240506 19:56:51.865056 12921 consensus_peers.cc:485] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:51.865633 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ce8749846847108edf5f554139cbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:51.865792 14249 consensus_queue.cc:1035] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:51.867998 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bcbf850d4914899a72f3b786d8b80b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.868243 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a94ecb64b4b64889b3585e64fff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.872831 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 867) took 1266 ms. Trace:
I20240506 19:56:51.872917 13243 rpcz_store.cc:276] 0506 19:56:50.606516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.606552 (+ 36us) service_pool.cc:225] Handling call
0506 19:56:51.872688 (+1266136us) ts_tablet_manager.cc:1120] Deleting tablet 4c2eb5dac9d34fb09119868bf5a85803
0506 19:56:51.872700 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.872811 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266010}
W20240506 19:56:51.873365 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4c2eb5dac9d34fb09119868bf5a85803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c2eb5dac9d34fb09119868bf5a85803 already in progress: creating tablet
I20240506 19:56:51.873729 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 868) took 1266 ms. Trace:
I20240506 19:56:51.873807 13243 rpcz_store.cc:276] 0506 19:56:50.606795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.606831 (+ 36us) service_pool.cc:225] Handling call
0506 19:56:51.873636 (+1266805us) ts_tablet_manager.cc:1120] Deleting tablet 1e64cb2f31b64b07848c2dfed18ab9f1
0506 19:56:51.873645 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.873712 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266678}
I20240506 19:56:51.874027 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a58ed1c4bd4f80834f9c6f52355334. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.874225 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e64cb2f31b64b07848c2dfed18ab9f1 already in progress: creating tablet
I20240506 19:56:51.874555 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 873) took 1243 ms. Trace:
I20240506 19:56:51.874620 13243 rpcz_store.cc:276] 0506 19:56:50.630938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.630992 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:51.874442 (+1243450us) ts_tablet_manager.cc:1120] Deleting tablet bd26e6b8115644cabe6532c7be1bbe95
0506 19:56:51.874451 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.874538 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1243201}
W20240506 19:56:51.875344 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bd26e6b8115644cabe6532c7be1bbe95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd26e6b8115644cabe6532c7be1bbe95 already in progress: creating tablet
I20240506 19:56:51.875597 13243 tablet_replica.cc:330] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:51.875767 13243 raft_consensus.cc:2229] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.875958 13243 raft_consensus.cc:2258] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.876499 13243 ts_tablet_manager.cc:1875] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:51.877537 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3141b13136c147b8b9d199c9728208e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.878722 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ce8749846847108edf5f554139cbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.882194 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f07ed7653f4910b3a47890842216bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.883986 13242 ts_tablet_manager.cc:1888] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:51.884094 13242 log.cc:1196] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/f3e7329b22414da886dee4e1ec6bf3ad
I20240506 19:56:51.884364 13242 ts_tablet_manager.cc:1909] T f3e7329b22414da886dee4e1ec6bf3ad P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:51.884672 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 748) took 2803 ms. Trace:
I20240506 19:56:51.886700 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a94ecb64b4b64889b3585e64fff5a (DEFAULT_TABLE table=default.Table_110baf1a2d08406abaddd6e76dd187a4 [id=7ce9557d82c14955b66d3970197b5bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.887215 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 57a58ed1c4bd4f80834f9c6f52355334 (DEFAULT_TABLE table=default.Table_4450908020fd475c8c535415af0a8925 [id=5c6305fd4cde4b1a87ae85c9267d759e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.887645 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ce8749846847108edf5f554139cbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:51.891338 14274 raft_consensus.cc:482] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.891491 14274 raft_consensus.cc:504] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.892161 14274 leader_election.cc:288] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.892560 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6e0126d9854215a0e174a76c0a0391" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:51.892730 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6e0126d9854215a0e174a76c0a0391" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:51.893337 12921 leader_election.cc:341] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.884764 13242 rpcz_store.cc:276] 0506 19:56:49.081311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.093041 (+ 11730us) service_pool.cc:225] Handling call
0506 19:56:51.684980 (+2591939us) ts_tablet_manager.cc:1120] Deleting tablet f3e7329b22414da886dee4e1ec6bf3ad
0506 19:56:51.684987 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.878468 (+193481us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.883911 (+ 5443us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:51.884651 (+ 740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2591753}
I20240506 19:56:51.898838 13249 tablet_bootstrap.cc:492] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:51.900960 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet f3e7329b22414da886dee4e1ec6bf3ad (table default.Table_e6fcb1ff22cd48b3a30306f27fe3c4cc [id=49660c755f8f470b8b8dc301ca27bb26]) successfully deleted
I20240506 19:56:51.904044 13249 ts_tablet_manager.cc:1367] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.003s
I20240506 19:56:51.905342 14211 raft_consensus.cc:482] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.905535 14211 raft_consensus.cc:504] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.906289 14211 leader_election.cc:288] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.906447 14211 raft_consensus.cc:482] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.906597 14211 raft_consensus.cc:504] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.906747 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 754) took 2780 ms. Trace:
I20240506 19:56:51.909510 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a94ecb64b4b64889b3585e64fff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.909776 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a58ed1c4bd4f80834f9c6f52355334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.909732 13120 raft_consensus.cc:2454] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:51.910115 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeae70e0d1144ea9e3e0f8727c15e6b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:51.910776 12922 leader_election.cc:302] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.911020 14249 raft_consensus.cc:2781] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:51.911090 14249 raft_consensus.cc:482] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:51.911149 14249 raft_consensus.cc:3037] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:51.911286 13057 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.913589 13245 tablet_bootstrap.cc:492] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:51.913692 14249 raft_consensus.cc:504] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.858914 14223 raft_consensus.cc:504] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.915686 14249 leader_election.cc:288] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:51.915547 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6e0126d9854215a0e174a76c0a0391" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:51.917917 13245 ts_tablet_manager.cc:1367] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.004s
I20240506 19:56:51.918484 13120 raft_consensus.cc:3037] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.922390 14223 leader_election.cc:288] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.922863 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c7c7b2da9e403f8241a512e210c4d9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:51.923060 12982 raft_consensus.cc:3037] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:51.924041 13245 raft_consensus.cc:348] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.924365 13245 raft_consensus.cc:374] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.924463 13245 raft_consensus.cc:729] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.924748 13245 consensus_queue.cc:260] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.925235 13245 ts_tablet_manager.cc:1398] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240506 19:56:51.925506 13245 tablet_bootstrap.cc:492] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:51.925750 12982 raft_consensus.cc:2454] T 07c7c7b2da9e403f8241a512e210c4d9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:51.928517 13120 raft_consensus.cc:2454] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:51.928992 12922 leader_election.cc:302] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:51.929267 14249 raft_consensus.cc:2781] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.930224 13245 tablet_bootstrap.cc:654] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:51.930629 13057 leader_election.cc:302] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:51.930752 14249 raft_consensus.cc:686] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:51.930912 14223 raft_consensus.cc:2781] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:51.931001 14223 raft_consensus.cc:686] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:51.931193 14223 consensus_queue.cc:237] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.931032 14249 consensus_queue.cc:237] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.932965 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 154e6d7c8f72480fa36fe542a5a67706 (DEFAULT_TABLE table=default.Table_3482045609cb42209a8222185144e10f [id=66cd489407394793a3016a8b41c2e62a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.933455 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154e6d7c8f72480fa36fe542a5a67706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.909907 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeae70e0d1144ea9e3e0f8727c15e6b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:51.914768 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6e0126d9854215a0e174a76c0a0391" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:51.941121 13242 rpcz_store.cc:276] 0506 19:56:49.125446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.125742 (+ 296us) service_pool.cc:225] Handling call
0506 19:56:51.903909 (+2778167us) ts_tablet_manager.cc:1120] Deleting tablet 79592a4ecacb4324b2278e1b9a75496c
0506 19:56:51.903919 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.905854 (+ 1935us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2777640}
I20240506 19:56:51.922113 14069 raft_consensus.cc:482] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.941579 14069 raft_consensus.cc:504] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:51.941752 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 79592a4ecacb4324b2278e1b9a75496c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79592a4ecacb4324b2278e1b9a75496c already in progress: creating tablet
I20240506 19:56:51.942477 14069 leader_election.cc:288] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.942632 14069 raft_consensus.cc:482] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:51.942754 14069 raft_consensus.cc:504] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.943467 14069 leader_election.cc:288] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:51.943496 13244 tablet_bootstrap.cc:492] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:51.922793 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c7c7b2da9e403f8241a512e210c4d9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:51.935370 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 154e6d7c8f72480fa36fe542a5a67706 (DEFAULT_TABLE table=default.Table_3482045609cb42209a8222185144e10f [id=66cd489407394793a3016a8b41c2e62a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.943897 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154e6d7c8f72480fa36fe542a5a67706. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:51.945232 12783 leader_election.cc:341] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d8f3e7df09f347bfa5cad4de34dbda13
W20240506 19:56:51.946066 13058 leader_election.cc:341] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:51.946390 13058 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:51.946462 13058 leader_election.cc:302] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:51.946456 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75467ef60504763a67fc10e8639716c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:51.955397 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_92e6b60423294d189484e6f67d5eb1ad" } modify_external_catalogs: true
I20240506 19:56:51.963220 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_92e6b60423294d189484e6f67d5eb1ad" } modify_external_catalogs: true
I20240506 19:56:51.945667 12816 tablet_service.cc:1467] Processing CreateTablet for tablet c727c9c20df34c4aa5c44b70add0d6b6 (DEFAULT_TABLE table=default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.963654 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c727c9c20df34c4aa5c44b70add0d6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.956142 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_515baa4fc8794cbf904ee3ed951e98b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.966495 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_515baa4fc8794cbf904ee3ed951e98b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.956418 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e9cb742b1de84e509d314bf1cc1444ed" } new_table_name: "default.Table_61becbd043ee470ca2ee3f4969d59971" modify_external_catalogs: true
I20240506 19:56:51.969023 12962 tablet_service.cc:1467] Processing CreateTablet for tablet c727c9c20df34c4aa5c44b70add0d6b6 (DEFAULT_TABLE table=default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:51.956661 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ff86f5a848874ce7960fbcc9c79457fd" } new_table_name: "default.Table_7c64e545a04644e798e5a0df2ca79723" modify_external_catalogs: true
I20240506 19:56:51.956899 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_32db184637f04319ae8841be3169f1e7" } modify_external_catalogs: true
I20240506 19:56:51.970552 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_32db184637f04319ae8841be3169f1e7" } modify_external_catalogs: true
I20240506 19:56:51.971983 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_52654f2617b946ad99d761a8a1b45c4d" } modify_external_catalogs: true
I20240506 19:56:51.972182 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_52654f2617b946ad99d761a8a1b45c4d" } modify_external_catalogs: true
I20240506 19:56:51.953752 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 753) took 2830 ms. Trace:
I20240506 19:56:51.942827 13249 raft_consensus.cc:348] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.974742 13242 rpcz_store.cc:276] 0506 19:56:49.122974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.126823 (+ 3849us) service_pool.cc:225] Handling call
0506 19:56:51.953562 (+2826739us) ts_tablet_manager.cc:1120] Deleting tablet 63f56d2180b1472a8168fb3a63589065
0506 19:56:51.953574 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.953725 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2788136}
I20240506 19:56:51.977106 13249 raft_consensus.cc:374] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:51.977723 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 63f56d2180b1472a8168fb3a63589065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63f56d2180b1472a8168fb3a63589065 already in progress: creating tablet
I20240506 19:56:51.953954 13244 ts_tablet_manager.cc:1367] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.212s user 0.005s sys 0.009s
I20240506 19:56:51.977958 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c727c9c20df34c4aa5c44b70add0d6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:51.978646 13249 raft_consensus.cc:729] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.979044 13242 tablet_replica.cc:330] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:51.979236 13242 raft_consensus.cc:2229] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:51.979377 13242 raft_consensus.cc:2258] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:51.980013 13244 raft_consensus.cc:348] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.980219 13244 raft_consensus.cc:374] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:51.980281 13244 raft_consensus.cc:729] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:51.982197 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 67e8b8be953a4a5784f2d098fd6364e3
W20240506 19:56:51.983634 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee69d8e09c2f4c898ce547ec396ca0e6 consensus state has no leader
I20240506 19:56:51.984776 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:51.985575 13249 consensus_queue.cc:260] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.986322 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_51d50ac0d42d4388a250ec5fd84128a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.986943 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51d50ac0d42d4388a250ec5fd84128a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.988133 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b2e9c53dcdb14928b637180cdca5f786"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.989131 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2e9c53dcdb14928b637180cdca5f786 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.989359 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8a9b92795a7742caa2ee99ab7ca138bf
I20240506 19:56:51.989655 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c30aa0d022c449e8b993bfa4e61fec2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:51.990254 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c30aa0d022c449e8b993bfa4e61fec2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.990201 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:51.990476 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9b92795a7742caa2ee99ab7ca138bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:51.991014 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:51.992520 13244 consensus_queue.cc:260] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:51.993144 13244 ts_tablet_manager.cc:1398] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240506 19:56:51.993467 13244 tablet_bootstrap.cc:492] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:51.994027 13249 ts_tablet_manager.cc:1398] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.088s user 0.001s sys 0.001s
I20240506 19:56:51.994342 13249 tablet_bootstrap.cc:492] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:51.995201 13242 ts_tablet_manager.cc:1875] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.001859 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.002081 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9b92795a7742caa2ee99ab7ca138bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.006315 14333 raft_consensus.cc:2726] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.007457 14333 raft_consensus.cc:482] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.008106 14333 raft_consensus.cc:504] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.009268 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet cd57563cc6c140c184d1725a2da6a475 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:52.009380 12921 leader_election.cc:341] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.010100 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26e6b8115644cabe6532c7be1bbe95" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.010551 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75467ef60504763a67fc10e8639716c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.010679 13249 tablet_bootstrap.cc:654] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.012274 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 910d1465df784dfba6d78ff7fd9f2722 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:52.009326 14333 leader_election.cc:288] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.013176 12781 leader_election.cc:302] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:52.013340 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4c8fb0fb09c4d32b8335fc6bd0827c7 consensus state has no leader
I20240506 19:56:52.013504 14069 raft_consensus.cc:2726] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.013743 14314 raft_consensus.cc:482] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.013876 14314 raft_consensus.cc:504] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:52.014243 13057 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.014498 14314 leader_election.cc:288] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:52.014477 12781 leader_election.cc:341] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d8f3e7df09f347bfa5cad4de34dbda13
I20240506 19:56:52.014631 12781 leader_election.cc:302] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:52.014942 14069 raft_consensus.cc:2726] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.015703 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a1401627948bbaff78fae4d42ab22" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.016207 12921 leader_election.cc:341] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.017196 13244 tablet_bootstrap.cc:654] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.017619 13246 ts_tablet_manager.cc:1888] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.017757 13246 log.cc:1196] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b899fb38bfed4e02b3708f6abec80e0a
I20240506 19:56:52.018083 13246 ts_tablet_manager.cc:1909] T b899fb38bfed4e02b3708f6abec80e0a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.018411 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 789) took 2880 ms. Trace:
I20240506 19:56:52.018501 13246 rpcz_store.cc:276] 0506 19:56:49.137753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.137816 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:51.770449 (+2632633us) ts_tablet_manager.cc:1120] Deleting tablet b899fb38bfed4e02b3708f6abec80e0a
0506 19:56:51.770458 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:51.849697 (+ 79239us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.017491 (+167794us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.018390 (+ 899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2632272}
I20240506 19:56:52.018949 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 790) took 2698 ms. Trace:
I20240506 19:56:52.019029 13246 rpcz_store.cc:276] 0506 19:56:49.320680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.321900 (+ 1220us) service_pool.cc:225] Handling call
0506 19:56:52.018823 (+2696923us) ts_tablet_manager.cc:1120] Deleting tablet f63e8e12fd3b453a96413f1fc769fb9f
0506 19:56:52.018832 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.018932 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2696585}
W20240506 19:56:52.019124 12921 consensus_peers.cc:485] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:52.019292 13246 tablet_replica.cc:330] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.019467 13246 raft_consensus.cc:2229] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.019655 13246 raft_consensus.cc:2258] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.020241 13246 ts_tablet_manager.cc:1875] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.021283 13245 tablet_bootstrap.cc:492] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.021452 13245 ts_tablet_manager.cc:1367] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.096s user 0.005s sys 0.018s
I20240506 19:56:52.022572 13245 raft_consensus.cc:348] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.022845 13245 raft_consensus.cc:374] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.022959 13245 raft_consensus.cc:729] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.023243 13245 consensus_queue.cc:260] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.023864 13245 ts_tablet_manager.cc:1398] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:52.024199 13245 tablet_bootstrap.cc:492] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.025629 13245 tablet_bootstrap.cc:654] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.027403 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b899fb38bfed4e02b3708f6abec80e0a (table default.Table_d6d662a23cfb4d02be80346ea62fbe2a [id=8dca79d6b027426c8f12402280fea23a]) successfully deleted
W20240506 19:56:52.027868 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f63e8e12fd3b453a96413f1fc769fb9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63e8e12fd3b453a96413f1fc769fb9f already in progress: creating tablet
W20240506 19:56:52.025951 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 79594us
I20240506 19:56:52.026800 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 154e6d7c8f72480fa36fe542a5a67706 (DEFAULT_TABLE table=default.Table_3482045609cb42209a8222185144e10f [id=66cd489407394793a3016a8b41c2e62a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.027303 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.027611 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9b92795a7742caa2ee99ab7ca138bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.028120 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.029575 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154e6d7c8f72480fa36fe542a5a67706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.029577 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26e6b8115644cabe6532c7be1bbe95" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.031355 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_74ef5084696549ca903c62d3a459320b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.032019 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ef5084696549ca903c62d3a459320b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.033274 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_73d6d75fc8f6456cba8ffeba3ecc9404" } new_table_name: "default.Table_b75f9589d33c4ca6b64d52bbf4bf7bca" modify_external_catalogs: true
I20240506 19:56:52.035208 14326 raft_consensus.cc:482] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.035455 14326 raft_consensus.cc:504] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.036072 14326 leader_election.cc:288] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.036253 14326 raft_consensus.cc:482] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.036355 14326 raft_consensus.cc:504] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.036556 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7c61244620487bbbac26dcdbf8a38c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:52.037228 13057 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.037819 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df3ab9dff5148f9bb335a7209b29bbc" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:52.038379 13057 leader_election.cc:341] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.038579 14333 raft_consensus.cc:482] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.038723 14333 raft_consensus.cc:504] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.039248 14333 leader_election.cc:288] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.039450 14333 raft_consensus.cc:482] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.039577 14333 raft_consensus.cc:504] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.040077 14333 leader_election.cc:288] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.040794 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0f74c7c4cd42d293fa244de3fd693b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.041337 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 597c61f1d1104256b90f6601cf3e6667 (DEFAULT_TABLE table=default.Table_715977f202b24ca7a07c82ef612e23bb [id=ef86771f67d14d9090bcc7722a834478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.041746 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 597c61f1d1104256b90f6601cf3e6667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.042943 13096 tablet_service.cc:1467] Processing CreateTablet for tablet c727c9c20df34c4aa5c44b70add0d6b6 (DEFAULT_TABLE table=default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.043409 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c727c9c20df34c4aa5c44b70add0d6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.044935 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab55564dcee4e6196671ac17d3b750c (DEFAULT_TABLE table=default.Table_c649504434cb4d69ac0d74733ff3591e [id=a3c712f1c0db496e908d90ae0af131ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.045272 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab55564dcee4e6196671ac17d3b750c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.045572 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab55564dcee4e6196671ac17d3b750c (DEFAULT_TABLE table=default.Table_c649504434cb4d69ac0d74733ff3591e [id=a3c712f1c0db496e908d90ae0af131ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.046011 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab55564dcee4e6196671ac17d3b750c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.047580 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 597c61f1d1104256b90f6601cf3e6667 (DEFAULT_TABLE table=default.Table_715977f202b24ca7a07c82ef612e23bb [id=ef86771f67d14d9090bcc7722a834478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.047962 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 597c61f1d1104256b90f6601cf3e6667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.048195 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c96f014293242be9640515697c0733b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:52.048745 13057 leader_election.cc:341] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:52.048946 13057 leader_election.cc:341] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.050858 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab55564dcee4e6196671ac17d3b750c (DEFAULT_TABLE table=default.Table_c649504434cb4d69ac0d74733ff3591e [id=a3c712f1c0db496e908d90ae0af131ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.051226 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab55564dcee4e6196671ac17d3b750c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.058247 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 597c61f1d1104256b90f6601cf3e6667 (DEFAULT_TABLE table=default.Table_715977f202b24ca7a07c82ef612e23bb [id=ef86771f67d14d9090bcc7722a834478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.058727 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 597c61f1d1104256b90f6601cf3e6667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.037206 14326 leader_election.cc:288] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.062736 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a1401627948bbaff78fae4d42ab22" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.062944 13246 ts_tablet_manager.cc:1888] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:52.062942 13120 raft_consensus.cc:2454] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.063091 13246 log.cc:1196] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/aae313961a914ed485f866d85785a788
I20240506 19:56:52.063432 13246 ts_tablet_manager.cc:1909] T aae313961a914ed485f866d85785a788 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.063753 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 791) took 2690 ms. Trace:
I20240506 19:56:52.063832 13246 rpcz_store.cc:276] 0506 19:56:49.373425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.373694 (+ 269us) service_pool.cc:225] Handling call
0506 19:56:52.019249 (+2645555us) ts_tablet_manager.cc:1120] Deleting tablet aae313961a914ed485f866d85785a788
0506 19:56:52.019256 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.039046 (+ 19790us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.058971 (+ 19925us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.063736 (+ 4765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2645324}
I20240506 19:56:52.064725 12922 leader_election.cc:302] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.065146 14314 raft_consensus.cc:2781] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.065228 14314 raft_consensus.cc:482] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.065312 14314 raft_consensus.cc:3037] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.067499 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7c61244620487bbbac26dcdbf8a38c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.067610 13246 tablet_replica.cc:330] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.067811 13246 raft_consensus.cc:2229] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.067900 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df3ab9dff5148f9bb335a7209b29bbc" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.067975 13246 raft_consensus.cc:2258] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.067896 14314 raft_consensus.cc:504] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.068145 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0f74c7c4cd42d293fa244de3fd693b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.068532 13246 ts_tablet_manager.cc:1875] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.069139 14314 leader_election.cc:288] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:52.070015 13058 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.070211 13058 leader_election.cc:302] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.070950 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a1401627948bbaff78fae4d42ab22" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.071126 13120 raft_consensus.cc:3037] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.074107 13120 raft_consensus.cc:2454] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.075381 13245 tablet_bootstrap.cc:492] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.075525 13245 ts_tablet_manager.cc:1367] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.000s
W20240506 19:56:52.076658 13058 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.076839 13058 leader_election.cc:302] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.077111 14333 raft_consensus.cc:2726] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.077031 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c96f014293242be9640515697c0733b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.072337 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a1401627948bbaff78fae4d42ab22" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.077651 14333 raft_consensus.cc:2726] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:52.078073 13058 leader_election.cc:341] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.078246 13058 leader_election.cc:302] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.077987 13245 raft_consensus.cc:348] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.078505 14333 raft_consensus.cc:2726] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.078595 13245 raft_consensus.cc:374] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.078892 13245 raft_consensus.cc:729] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:52.079062 13058 leader_election.cc:341] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.079201 13058 leader_election.cc:302] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.079438 14333 raft_consensus.cc:2726] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:52.079749 13058 leader_election.cc:341] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.079890 13058 leader_election.cc:302] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.080106 14333 raft_consensus.cc:2726] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.079130 13245 consensus_queue.cc:260] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.080670 13245 ts_tablet_manager.cc:1398] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:52.080962 13245 tablet_bootstrap.cc:492] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.083108 12516 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet aa0b7e07a3c040f1bc6ac2428f308eec
W20240506 19:56:52.083557 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ace0ec43ba44060bcec2a50e1f48098 consensus state has no leader
I20240506 19:56:52.084015 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 35198d5786c14de89c9751a0372ba91d
W20240506 19:56:52.084537 13211 log.cc:924] Time spent T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240506 19:56:52.086304 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 35198d5786c14de89c9751a0372ba91d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.090210 13243 ts_tablet_manager.cc:1888] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.090384 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]) successfully deleted
I20240506 19:56:52.090605 13243 log.cc:1196] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b3e50bde3564493eb7267acb42cb8592
I20240506 19:56:52.071760 14331 raft_consensus.cc:482] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.091274 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 35198d5786c14de89c9751a0372ba91d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.091326 14331 raft_consensus.cc:504] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.093879 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0eb81c2cf9b549a28daebf834d26613c" } new_table_name: "default.Table_91e0d90fa1a442a2a442d06baf1ef73f" modify_external_catalogs: true
I20240506 19:56:52.094705 14331 leader_election.cc:288] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.096848 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6a081bfc6c694bd2bb37cd06213e221a" } modify_external_catalogs: true
I20240506 19:56:52.096932 12922 leader_election.cc:302] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:52.097081 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6a081bfc6c694bd2bb37cd06213e221a" } modify_external_catalogs: true
W20240506 19:56:52.097360 12921 leader_election.cc:341] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.098192 14314 raft_consensus.cc:2781] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.098309 14314 raft_consensus.cc:686] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.098398 14274 consensus_queue.cc:1035] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:52.098841 12783 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: da41ef03182445f1946584c2bf015139
I20240506 19:56:52.099045 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_742be09ecdef4a4c9d11965a17109e81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.099865 12922 consensus_peers.cc:485] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b899fb38bfed4e02b3708f6abec80e0a. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:52.101963 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_742be09ecdef4a4c9d11965a17109e81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.103313 13245 tablet_bootstrap.cc:654] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.103808 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6fb4b0e1d5c94d88b7f9792005908344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.104406 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fb4b0e1d5c94d88b7f9792005908344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.105027 13244 tablet_bootstrap.cc:492] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:52.105209 13244 ts_tablet_manager.cc:1367] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.112s user 0.005s sys 0.009s
I20240506 19:56:52.105983 13244 raft_consensus.cc:348] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.106313 13244 raft_consensus.cc:374] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.106426 13244 raft_consensus.cc:729] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.112212 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 35198d5786c14de89c9751a0372ba91d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.098533 14314 consensus_queue.cc:237] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.122892 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 9956ca4948644237a768806d94873c82 (DEFAULT_TABLE table=default.Table_84cf0c067fa84ec2b9b35f2f981d4671 [id=8f48c6efec1c40b78353ad7d31c92096]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:52.109408 12781 leader_election.cc:341] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: da41ef03182445f1946584c2bf015139
I20240506 19:56:52.124658 12781 leader_election.cc:302] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:52.124842 13243 ts_tablet_manager.cc:1909] T b3e50bde3564493eb7267acb42cb8592 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.124945 14331 raft_consensus.cc:2726] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.126318 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 882) took 1261 ms. Trace:
I20240506 19:56:52.122602 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 9956ca4948644237a768806d94873c82 (DEFAULT_TABLE table=default.Table_84cf0c067fa84ec2b9b35f2f981d4671 [id=8f48c6efec1c40b78353ad7d31c92096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.126672 13243 rpcz_store.cc:276] 0506 19:56:50.865309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.865359 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:51.875553 (+1010194us) ts_tablet_manager.cc:1120] Deleting tablet b3e50bde3564493eb7267acb42cb8592
0506 19:56:51.875562 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.057592 (+182030us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.090064 (+ 32472us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.126296 (+ 36232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":969127}
I20240506 19:56:52.127213 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 883) took 1261 ms. Trace:
I20240506 19:56:52.127279 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b3e50bde3564493eb7267acb42cb8592 (table default.Table_95be5bd2c51b4613b0152034aae6350d [id=b8f3e6ecf02f4dd299adfa1d7464e53f]) successfully deleted
I20240506 19:56:52.127429 13243 rpcz_store.cc:276] 0506 19:56:50.865633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.865724 (+ 91us) service_pool.cc:225] Handling call
0506 19:56:52.127084 (+1261360us) ts_tablet_manager.cc:1120] Deleting tablet 7f8989f02f25418ea9f824b3e5eeddd6
0506 19:56:52.127094 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.127193 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219695}
W20240506 19:56:52.126375 12921 consensus_peers.cc:485] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:52.127907 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7f8989f02f25418ea9f824b3e5eeddd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f8989f02f25418ea9f824b3e5eeddd6 already in progress: creating tablet
W20240506 19:56:52.128131 12921 consensus_peers.cc:485] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:52.128432 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:52.129877 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 885) took 1263 ms. Trace:
I20240506 19:56:52.130029 13243 rpcz_store.cc:276] 0506 19:56:50.865888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.865940 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:52.127884 (+1261944us) ts_tablet_manager.cc:1120] Deleting tablet 5b15189e5cb54684a0f62308c2e5f131
0506 19:56:52.127894 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.129603 (+ 1709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219930}
I20240506 19:56:52.124744 13244 consensus_queue.cc:260] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:52.131269 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5b15189e5cb54684a0f62308c2e5f131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b15189e5cb54684a0f62308c2e5f131 already in progress: creating tablet
I20240506 19:56:52.131889 13243 tablet_replica.cc:330] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.132124 13243 raft_consensus.cc:2229] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.133080 13243 pending_rounds.cc:62] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:52.133203 13243 pending_rounds.cc:68] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744083874381824 op_type: NO_OP noop_request { }
I20240506 19:56:52.133404 13243 raft_consensus.cc:2866] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:52.133540 13243 raft_consensus.cc:2258] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.134189 13243 ts_tablet_manager.cc:1875] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.138247 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 910d1465df784dfba6d78ff7fd9f2722
I20240506 19:56:52.138782 12515 catalog_manager.cc:6554] ReplaceTablet: tablet 910d1465df784dfba6d78ff7fd9f2722 deleted and replaced by tablet 01f0def08b17403180b9b876c26428f7
I20240506 19:56:52.139289 13244 ts_tablet_manager.cc:1398] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
W20240506 19:56:52.139312 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20d071e027174b03b01cf788c0b95e89 consensus state has no leader
I20240506 19:56:52.140249 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 910d1465df784dfba6d78ff7fd9f2722 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f0def08b17403180b9b876c26428f7) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.143301 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 910d1465df784dfba6d78ff7fd9f2722 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f0def08b17403180b9b876c26428f7) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.146553 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 910d1465df784dfba6d78ff7fd9f2722 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f0def08b17403180b9b876c26428f7) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.148627 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 9956ca4948644237a768806d94873c82 (DEFAULT_TABLE table=default.Table_84cf0c067fa84ec2b9b35f2f981d4671 [id=8f48c6efec1c40b78353ad7d31c92096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:52.150977 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9956ca4948644237a768806d94873c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.153478 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9956ca4948644237a768806d94873c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.159291 13249 tablet_bootstrap.cc:492] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.159392 13243 ts_tablet_manager.cc:1888] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240506 19:56:52.161836 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet ee69d8e09c2f4c898ce547ec396ca0e6 (table default.Table_e300dc01d2fc4eb1bb7c22f60f49f570 [id=38df61b9f4a641eeb68e44b1eadf7d1a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ee69d8e09c2f4c898ce547ec396ca0e6 consensus state has no leader
I20240506 19:56:52.168402 14223 raft_consensus.cc:482] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.168637 14223 raft_consensus.cc:504] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.169508 14223 leader_election.cc:288] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.170609 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.171324 13058 leader_election.cc:341] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.172534 13243 log.cc:1196] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/9bb90bf767c94d73ace6411cfd31d65c
I20240506 19:56:52.170814 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.173239 13243 ts_tablet_manager.cc:1909] T 9bb90bf767c94d73ace6411cfd31d65c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:52.173511 13057 leader_election.cc:341] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.173719 13245 tablet_bootstrap.cc:492] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.173671 13057 leader_election.cc:302] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.173880 13245 ts_tablet_manager.cc:1367] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20240506 19:56:52.174281 14223 raft_consensus.cc:2726] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.159446 13249 ts_tablet_manager.cc:1367] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.165s user 0.001s sys 0.013s
I20240506 19:56:52.174711 13245 raft_consensus.cc:348] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.174916 13245 raft_consensus.cc:374] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.174984 13245 raft_consensus.cc:729] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.175212 13245 consensus_queue.cc:260] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.175243 13249 raft_consensus.cc:348] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.175519 13249 raft_consensus.cc:374] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.175611 13249 raft_consensus.cc:729] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.175869 13245 ts_tablet_manager.cc:1398] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:52.175875 13249 consensus_queue.cc:260] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.176147 13245 tablet_bootstrap.cc:492] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.178215 13244 tablet_bootstrap.cc:492] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:52.179833 13244 tablet_bootstrap.cc:654] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.181650 14343 tablet.cc:1722] T 176c61208b1c4518820a829d9f5c0a6f P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:52.183418 13249 ts_tablet_manager.cc:1398] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:52.183699 13249 tablet_bootstrap.cc:492] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.185599 13246 ts_tablet_manager.cc:1888] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.185722 13246 log.cc:1196] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f4e4be565e2d4c59b3587baed72e4a9c
I20240506 19:56:52.186100 13246 ts_tablet_manager.cc:1909] T f4e4be565e2d4c59b3587baed72e4a9c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.186110 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9956ca4948644237a768806d94873c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:52.186434 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 792) took 2759 ms. Trace:
I20240506 19:56:52.186513 13246 rpcz_store.cc:276] 0506 19:56:49.427175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.427326 (+ 151us) service_pool.cc:225] Handling call
0506 19:56:52.067543 (+2640217us) ts_tablet_manager.cc:1120] Deleting tablet f4e4be565e2d4c59b3587baed72e4a9c
0506 19:56:52.067560 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.114046 (+ 46486us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.185485 (+ 71439us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.186416 (+ 931us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2636571}
I20240506 19:56:52.186839 13246 tablet_replica.cc:330] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.187057 13246 raft_consensus.cc:2229] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.187216 13246 raft_consensus.cc:2258] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.187742 13246 ts_tablet_manager.cc:1875] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.189667 13249 tablet_bootstrap.cc:654] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.190122 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 884) took 1324 ms. Trace:
I20240506 19:56:52.192873 13243 rpcz_store.cc:276] 0506 19:56:50.865800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.866163 (+ 363us) service_pool.cc:225] Handling call
0506 19:56:52.131619 (+1265456us) ts_tablet_manager.cc:1120] Deleting tablet 9bb90bf767c94d73ace6411cfd31d65c
0506 19:56:52.131846 (+ 227us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.150278 (+ 18432us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.159297 (+ 9019us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.190098 (+ 30801us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223655}
I20240506 19:56:52.193651 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 9bb90bf767c94d73ace6411cfd31d65c (table default.Table_a410467878c2481f86118d12ccd06f28 [id=f339c5ce6df04d839431b2475a6c1d8d]) successfully deleted
I20240506 19:56:52.194326 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 886) took 1328 ms. Trace:
I20240506 19:56:52.194919 13243 rpcz_store.cc:276] 0506 19:56:50.865984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.869008 (+ 3024us) service_pool.cc:225] Handling call
0506 19:56:52.194184 (+1325176us) ts_tablet_manager.cc:1120] Deleting tablet 996eb97639464e0bb976f2b1ecbbb73b
0506 19:56:52.194195 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.194305 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286169}
W20240506 19:56:52.195780 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 996eb97639464e0bb976f2b1ecbbb73b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996eb97639464e0bb976f2b1ecbbb73b already in progress: creating tablet
I20240506 19:56:52.196318 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 881) took 1331 ms. Trace:
I20240506 19:56:52.197001 13243 rpcz_store.cc:276] 0506 19:56:50.865187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.893645 (+ 28458us) service_pool.cc:225] Handling call
0506 19:56:52.196181 (+1302536us) ts_tablet_manager.cc:1120] Deleting tablet a03f574aec984b83ab596e92a6f323fc
0506 19:56:52.196191 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.196299 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287364}
I20240506 19:56:52.197861 13245 tablet_bootstrap.cc:654] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:52.198150 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a03f574aec984b83ab596e92a6f323fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a03f574aec984b83ab596e92a6f323fc already in progress: creating tablet
I20240506 19:56:52.199136 14342 tablet.cc:1722] T 176c61208b1c4518820a829d9f5c0a6f P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:52.207425 13245 tablet_bootstrap.cc:492] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.207590 13245 ts_tablet_manager.cc:1367] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20240506 19:56:52.208365 13245 raft_consensus.cc:348] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.208570 13245 raft_consensus.cc:374] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.208621 13245 raft_consensus.cc:729] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.208824 13245 consensus_queue.cc:260] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.220705 13243 tablet_replica.cc:330] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.221482 13243 raft_consensus.cc:2229] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.221922 13243 raft_consensus.cc:2258] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.226270 13245 ts_tablet_manager.cc:1398] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240506 19:56:52.226454 13243 ts_tablet_manager.cc:1875] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.229807 13245 tablet_bootstrap.cc:492] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.230274 13249 tablet_bootstrap.cc:492] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.230552 13249 ts_tablet_manager.cc:1367] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20240506 19:56:52.232188 13245 tablet_bootstrap.cc:654] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.233983 13249 raft_consensus.cc:348] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.234262 13249 raft_consensus.cc:374] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.234364 13249 raft_consensus.cc:729] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.234656 13249 consensus_queue.cc:260] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.235885 13246 ts_tablet_manager.cc:1888] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.236012 13246 log.cc:1196] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1a6e0126d9854215a0e174a76c0a0391
I20240506 19:56:52.236346 13246 ts_tablet_manager.cc:1909] T 1a6e0126d9854215a0e174a76c0a0391 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.236346 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f4e4be565e2d4c59b3587baed72e4a9c (table default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]) successfully deleted
I20240506 19:56:52.236702 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 793) took 2723 ms. Trace:
I20240506 19:56:52.236785 13246 rpcz_store.cc:276] 0506 19:56:49.513451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.513815 (+ 364us) service_pool.cc:225] Handling call
0506 19:56:52.186794 (+2672979us) ts_tablet_manager.cc:1120] Deleting tablet 1a6e0126d9854215a0e174a76c0a0391
0506 19:56:52.186804 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.214265 (+ 27461us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.235762 (+ 21497us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.236682 (+ 920us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816,"tablet-delete.queue_time_us":2638564}
I20240506 19:56:52.237159 13246 tablet_replica.cc:330] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.240020 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c44481b32b644df187cba705ae9759c5" } modify_external_catalogs: true
I20240506 19:56:52.240103 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f9f1fa97c6434970b181f7536994782e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.240736 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9f1fa97c6434970b181f7536994782e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.250609 13249 ts_tablet_manager.cc:1398] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20240506 19:56:52.257594 14331 raft_consensus.cc:482] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.258090 14331 raft_consensus.cc:504] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.258921 14331 leader_election.cc:288] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.259168 13249 tablet_bootstrap.cc:492] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:52.260370 12783 leader_election.cc:341] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: cd57563cc6c140c184d1725a2da6a475
W20240506 19:56:52.260506 12781 leader_election.cc:341] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: cd57563cc6c140c184d1725a2da6a475
I20240506 19:56:52.260622 12781 leader_election.cc:302] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:52.261049 14331 raft_consensus.cc:2726] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.261730 13244 tablet_bootstrap.cc:492] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:52.262265 13244 ts_tablet_manager.cc:1367] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.006s
I20240506 19:56:52.270956 13246 raft_consensus.cc:2229] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.271777 13246 pending_rounds.cc:62] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:52.274338 13249 tablet_bootstrap.cc:654] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.279356 13246 pending_rounds.cc:68] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744080667009024 op_type: NO_OP noop_request { }
I20240506 19:56:52.279556 13246 raft_consensus.cc:2866] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:52.279660 13246 raft_consensus.cc:2258] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.280328 13246 ts_tablet_manager.cc:1875] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.283171 13245 tablet_bootstrap.cc:492] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.283319 13245 ts_tablet_manager.cc:1367] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.001s
I20240506 19:56:52.284145 13245 raft_consensus.cc:348] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.284365 13245 raft_consensus.cc:374] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.284438 13245 raft_consensus.cc:729] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.284663 13245 consensus_queue.cc:260] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.285923 13245 ts_tablet_manager.cc:1398] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:52.288290 13245 tablet_bootstrap.cc:492] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.290143 13245 tablet_bootstrap.cc:654] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.290052 13243 ts_tablet_manager.cc:1888] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.290671 13243 log.cc:1196] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/63f56d2180b1472a8168fb3a63589065
I20240506 19:56:52.291071 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6de4fcdd30e14b65ac04120e9e628710" } new_table_name: "default.Table_e33d6f2f5f404ec6be38fd97576e41ad" modify_external_catalogs: true
I20240506 19:56:52.292898 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fb883920a7c94fd4a786a7b48237c418" } modify_external_catalogs: true
I20240506 19:56:52.299072 13243 ts_tablet_manager.cc:1909] T 63f56d2180b1472a8168fb3a63589065 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.301628 13249 tablet_bootstrap.cc:492] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.301858 13249 ts_tablet_manager.cc:1367] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20240506 19:56:52.302654 13249 raft_consensus.cc:348] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.302771 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 887) took 1350 ms. Trace:
I20240506 19:56:52.302975 13249 raft_consensus.cc:374] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.303135 13249 raft_consensus.cc:729] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.303400 13249 consensus_queue.cc:260] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.303056 13243 rpcz_store.cc:276] 0506 19:56:50.952724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.952787 (+ 63us) service_pool.cc:225] Handling call
0506 19:56:52.198601 (+1245814us) ts_tablet_manager.cc:1120] Deleting tablet 63f56d2180b1472a8168fb3a63589065
0506 19:56:52.220638 (+ 22037us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.270332 (+ 49694us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.287180 (+ 16848us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.302660 (+ 15480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245579}
I20240506 19:56:52.304157 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 63f56d2180b1472a8168fb3a63589065 (table default.Table_ab69ac48aa7a4934a9f383276277cf10 [id=493f88f528114491b898c4a1156712fc]) successfully deleted
I20240506 19:56:52.266777 13244 raft_consensus.cc:348] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.304759 13246 ts_tablet_manager.cc:1888] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.305141 13243 tablet_replica.cc:330] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.305752 13249 ts_tablet_manager.cc:1398] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:52.306114 13249 tablet_bootstrap.cc:492] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.307866 13249 tablet_bootstrap.cc:654] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.307746 13243 raft_consensus.cc:2229] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.308909 13243 pending_rounds.cc:62] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:52.309264 13243 pending_rounds.cc:68] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744083545870336 op_type: NO_OP noop_request { }
I20240506 19:56:52.309808 13243 raft_consensus.cc:2866] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:52.313935 13246 log.cc:1196] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f8951bf4873e4382bcd42a52368fb376
I20240506 19:56:52.314374 13246 ts_tablet_manager.cc:1909] T f8951bf4873e4382bcd42a52368fb376 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.315961 13245 tablet_bootstrap.cc:492] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.316123 13245 ts_tablet_manager.cc:1367] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240506 19:56:52.316943 13245 raft_consensus.cc:348] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.317135 13249 tablet_bootstrap.cc:492] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.317162 13245 raft_consensus.cc:374] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.317232 13245 raft_consensus.cc:729] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.317438 13249 ts_tablet_manager.cc:1367] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240506 19:56:52.317479 13245 consensus_queue.cc:260] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.318171 13245 ts_tablet_manager.cc:1398] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:52.318358 13249 raft_consensus.cc:348] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.318593 13249 raft_consensus.cc:374] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.318686 13249 raft_consensus.cc:729] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.318948 13249 consensus_queue.cc:260] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.319576 13249 ts_tablet_manager.cc:1398] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:52.320173 13249 tablet_bootstrap.cc:492] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.321820 13249 tablet_bootstrap.cc:654] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.305014 13244 raft_consensus.cc:374] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.324447 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 794) took 2810 ms. Trace:
I20240506 19:56:52.324469 14223 raft_consensus.cc:482] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.324644 14223 raft_consensus.cc:504] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.325326 14223 leader_election.cc:288] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.325906 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb357a7eb2147a79cc69d024bd35269" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.326874 13245 tablet_bootstrap.cc:492] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:52.327203 13058 leader_election.cc:341] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.328665 13245 tablet_bootstrap.cc:654] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.329490 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb357a7eb2147a79cc69d024bd35269" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:52.330106 13057 leader_election.cc:341] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.330220 13057 leader_election.cc:302] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.330657 14223 raft_consensus.cc:2726] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:52.333465 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 779534d888ab4c41bae0d7fe5fffc390 consensus state has no leader
I20240506 19:56:52.337445 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fb883920a7c94fd4a786a7b48237c418" } modify_external_catalogs: true
I20240506 19:56:52.342096 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5fe8d4891c484627ac729971ff7aad3e
I20240506 19:56:52.343178 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8d4891c484627ac729971ff7aad3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.344329 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8d4891c484627ac729971ff7aad3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.324564 13246 rpcz_store.cc:276] 0506 19:56:49.514012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.514060 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:52.237108 (+2723048us) ts_tablet_manager.cc:1120] Deleting tablet f8951bf4873e4382bcd42a52368fb376
0506 19:56:52.237120 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.296220 (+ 59100us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.304631 (+ 8411us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.324420 (+ 19789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2688776}
I20240506 19:56:52.344681 12515 catalog_manager.cc:5484] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:52.345140 13246 tablet_replica.cc:330] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.345368 13246 raft_consensus.cc:2229] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.345374 12515 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]). Expected version 2 got 0
I20240506 19:56:52.345518 13246 raft_consensus.cc:2258] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.346203 13246 ts_tablet_manager.cc:1875] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.347332 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c44481b32b644df187cba705ae9759c5" } modify_external_catalogs: true
I20240506 19:56:52.350284 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5a46b91cb742a994496648fbf87aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.354116 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0bfd44a2bbd8485c82c64432fff1c61b" } new_table_name: "default.Table_0bcaa070661f4dc28a869fe16f7bc6c0" modify_external_catalogs: true
I20240506 19:56:52.356712 14080 raft_consensus.cc:482] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.356884 14080 raft_consensus.cc:504] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.357532 14080 leader_election.cc:288] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.357558 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_33e9d54058954ad9806125313b789621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.358122 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33e9d54058954ad9806125313b789621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.358175 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c81ac92d04ddc93cc52e5c01cbc60" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.358964 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c81ac92d04ddc93cc52e5c01cbc60" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.359123 12922 leader_election.cc:341] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: Tablet not RUNNING: STOPPED
W20240506 19:56:52.359469 12921 leader_election.cc:341] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.359553 12921 leader_election.cc:302] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.360220 14080 raft_consensus.cc:2726] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.360244 13246 ts_tablet_manager.cc:1888] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.360340 13246 log.cc:1196] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/bd2c81ac92d04ddc93cc52e5c01cbc60
I20240506 19:56:52.360638 13246 ts_tablet_manager.cc:1909] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.362941 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 795) took 2848 ms. Trace:
I20240506 19:56:52.363083 13246 rpcz_store.cc:276] 0506 19:56:49.514217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.514288 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:52.345078 (+2830790us) ts_tablet_manager.cc:1120] Deleting tablet bd2c81ac92d04ddc93cc52e5c01cbc60
0506 19:56:52.345093 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.354050 (+ 8957us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.360129 (+ 6079us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.362919 (+ 2790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2796646}
I20240506 19:56:52.363656 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 796) took 2837 ms. Trace:
I20240506 19:56:52.363755 13246 rpcz_store.cc:276] 0506 19:56:49.526173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.526225 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:52.363534 (+2837309us) ts_tablet_manager.cc:1120] Deleting tablet 3fb398132c58489fb0a28492302e4a23
0506 19:56:52.363545 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.363640 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2814434}
I20240506 19:56:52.368564 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 797) took 2834 ms. Trace:
I20240506 19:56:52.368667 13246 rpcz_store.cc:276] 0506 19:56:49.533887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.533942 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:52.368424 (+2834482us) ts_tablet_manager.cc:1120] Deleting tablet 58c88f80f5b34672b7c3a4b604469bf1
0506 19:56:52.368437 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.368543 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2819234}
I20240506 19:56:52.369197 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 803) took 2763 ms. Trace:
I20240506 19:56:52.369267 13246 rpcz_store.cc:276] 0506 19:56:49.605656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.641743 (+ 36087us) service_pool.cc:225] Handling call
0506 19:56:52.369075 (+2727332us) ts_tablet_manager.cc:1120] Deleting tablet 979d789bc89a435b9e6cd8d6dc9c2037
0506 19:56:52.369086 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.369183 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2726881}
I20240506 19:56:52.369928 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 806) took 2721 ms. Trace:
I20240506 19:56:52.369999 13246 rpcz_store.cc:276] 0506 19:56:49.648276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.648482 (+ 206us) service_pool.cc:225] Handling call
0506 19:56:52.369831 (+2721349us) ts_tablet_manager.cc:1120] Deleting tablet 744864043f9347e0a13f8c2a7d6b147f
0506 19:56:52.369840 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.369913 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2721112}
I20240506 19:56:52.370394 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 822) took 2520 ms. Trace:
I20240506 19:56:52.370462 13246 rpcz_store.cc:276] 0506 19:56:49.849722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.853704 (+ 3982us) service_pool.cc:225] Handling call
0506 19:56:52.370292 (+2516588us) ts_tablet_manager.cc:1120] Deleting tablet f3bbf5a4326448d684fcb3f138dcf96d
0506 19:56:52.370300 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.370379 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2516358}
I20240506 19:56:52.371023 13246 tablet_replica.cc:330] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.371213 13246 raft_consensus.cc:2229] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.371410 13246 raft_consensus.cc:2258] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.371604 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c047db21c5bb45a8a1a20ce3ade15da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:52.372032 13246 ts_tablet_manager.cc:1875] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:52.372252 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c047db21c5bb45a8a1a20ce3ade15da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.377199 13243 raft_consensus.cc:2258] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.381098 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3a9d0dcf4191468da0f575d88f4a47f4" } modify_external_catalogs: true
I20240506 19:56:52.381521 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3a9d0dcf4191468da0f575d88f4a47f4" } modify_external_catalogs: true
I20240506 19:56:52.384025 13249 tablet_bootstrap.cc:492] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.384178 13249 ts_tablet_manager.cc:1367] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.002s
I20240506 19:56:52.384886 13249 raft_consensus.cc:348] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.385067 13249 raft_consensus.cc:374] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.385107 13249 raft_consensus.cc:729] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.385314 13249 consensus_queue.cc:260] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.385931 13249 ts_tablet_manager.cc:1398] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:52.389220 13249 tablet_bootstrap.cc:492] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.390615 13249 tablet_bootstrap.cc:654] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.392201 12845 raft_consensus.cc:3037] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.397063 13243 ts_tablet_manager.cc:1875] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.324278 13244 raft_consensus.cc:729] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.405119 13244 consensus_queue.cc:260] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.406296 13244 ts_tablet_manager.cc:1398] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.144s user 0.002s sys 0.004s
I20240506 19:56:52.406782 13244 tablet_bootstrap.cc:492] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:52.416715 13249 tablet_bootstrap.cc:492] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.416896 13249 ts_tablet_manager.cc:1367] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
I20240506 19:56:52.417330 13246 ts_tablet_manager.cc:1888] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.417533 13246 log.cc:1196] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1615f6167aae4dc096b9f563ab5e6215
I20240506 19:56:52.417589 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1a6e0126d9854215a0e174a76c0a0391 (table default.Table_826f8c31905e4a76b9bd194c717a5853 [id=577dd46fb1b44fd380574a6cd0ba2695]) successfully deleted
I20240506 19:56:52.417636 13249 raft_consensus.cc:348] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.417876 13249 raft_consensus.cc:374] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.417932 13249 raft_consensus.cc:729] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.417994 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f8951bf4873e4382bcd42a52368fb376 (table default.Table_6974ae6398ae4559a50fb236133424b1 [id=07a119f6e5854b1090f2f47dc8c9dada]) successfully deleted
I20240506 19:56:52.418386 13246 ts_tablet_manager.cc:1909] T 1615f6167aae4dc096b9f563ab5e6215 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.420684 13249 consensus_queue.cc:260] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.426800 13249 ts_tablet_manager.cc:1398] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:52.426851 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e90b93023e40b3bafbddf7bbcc10b2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.427373 13058 leader_election.cc:341] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.428431 13249 tablet_bootstrap.cc:492] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.429884 13249 tablet_bootstrap.cc:654] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.430102 14211 leader_election.cc:288] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.430333 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8d4891c484627ac729971ff7aad3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.430725 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e90b93023e40b3bafbddf7bbcc10b2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.430851 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
W20240506 19:56:52.431319 13057 leader_election.cc:341] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.431428 13057 leader_election.cc:302] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.431694 14211 raft_consensus.cc:2726] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.418285 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet bd2c81ac92d04ddc93cc52e5c01cbc60 (table default.Table_ca6325447fde4d198454d0ac20e928bd [id=a5b2fb638e5a4ee2abfd610cb6274767]) successfully deleted
I20240506 19:56:52.432677 14080 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.432796 14080 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.432740 13244 tablet_bootstrap.cc:654] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.433449 14080 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:52.433663 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 3fb398132c58489fb0a28492302e4a23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb398132c58489fb0a28492302e4a23 already in progress: creating tablet
I20240506 19:56:52.434113 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.434296 13120 raft_consensus.cc:2454] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.434973 12922 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:52.435194 14274 raft_consensus.cc:482] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.435323 14274 raft_consensus.cc:504] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.435811 14274 leader_election.cc:288] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.435938 14274 raft_consensus.cc:2781] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 19:56:52.435961 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 827) took 2512 ms. Trace:
I20240506 19:56:52.435999 14274 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.436057 14274 raft_consensus.cc:3037] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Advancing to term 2
I20240506 19:56:52.436041 13246 rpcz_store.cc:276] 0506 19:56:49.917824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.917879 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:52.370749 (+2452870us) ts_tablet_manager.cc:1120] Deleting tablet 1615f6167aae4dc096b9f563ab5e6215
0506 19:56:52.370986 (+ 237us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.399245 (+ 28259us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.417186 (+ 17941us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.429926 (+ 12740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2452585}
I20240506 19:56:52.436407 13246 tablet_replica.cc:330] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.436592 13246 raft_consensus.cc:2229] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.436714 13246 raft_consensus.cc:2258] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:52.436604 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 58c88f80f5b34672b7c3a4b604469bf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58c88f80f5b34672b7c3a4b604469bf1 already in progress: creating tablet
I20240506 19:56:52.437367 13246 ts_tablet_manager.cc:1875] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:52.438230 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 979d789bc89a435b9e6cd8d6dc9c2037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 979d789bc89a435b9e6cd8d6dc9c2037 already in progress: creating tablet
I20240506 19:56:52.438573 14274 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.440203 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.440203 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.440559 13243 ts_tablet_manager.cc:1888] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240506 19:56:52.439450 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 744864043f9347e0a13f8c2a7d6b147f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 744864043f9347e0a13f8c2a7d6b147f already in progress: creating tablet
I20240506 19:56:52.441041 13243 log.cc:1196] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b899fb38bfed4e02b3708f6abec80e0a
I20240506 19:56:52.441207 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.441429 13120 raft_consensus.cc:2454] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:52.441617 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f3bbf5a4326448d684fcb3f138dcf96d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bbf5a4326448d684fcb3f138dcf96d already in progress: creating tablet
W20240506 19:56:52.441735 12921 leader_election.cc:341] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:52.441927 12921 leader_election.cc:341] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.442304 12922 leader_election.cc:302] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.442569 14314 raft_consensus.cc:2781] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.442643 14314 raft_consensus.cc:482] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.442706 14314 raft_consensus.cc:3037] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.442811 14274 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.443327 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.443495 13120 raft_consensus.cc:3037] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Advancing to term 2
I20240506 19:56:52.444481 13243 ts_tablet_manager.cc:1909] T b899fb38bfed4e02b3708f6abec80e0a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.445231 14314 raft_consensus.cc:504] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.446076 13120 raft_consensus.cc:2454] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 2.
I20240506 19:56:52.446523 12922 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.445232 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 888) took 1491 ms. Trace:
I20240506 19:56:52.446979 13243 rpcz_store.cc:276] 0506 19:56:50.953525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.953589 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:52.304971 (+1351382us) ts_tablet_manager.cc:1120] Deleting tablet b899fb38bfed4e02b3708f6abec80e0a
0506 19:56:52.304989 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.410616 (+105627us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.420838 (+ 10222us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.445212 (+ 24374us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1937,"tablet-delete.queue_time_us":1351012}
I20240506 19:56:52.447355 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.447499 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.447705 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b899fb38bfed4e02b3708f6abec80e0a (table default.Table_d6d662a23cfb4d02be80346ea62fbe2a [id=8dca79d6b027426c8f12402280fea23a]) successfully deleted
W20240506 19:56:52.447940 12921 leader_election.cc:341] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:52.448127 12921 leader_election.cc:341] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.448217 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 912) took 1261 ms. Trace:
I20240506 19:56:52.446575 14314 leader_election.cc:288] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.448447 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.448565 14314 raft_consensus.cc:2781] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 19:56:52.448679 13120 raft_consensus.cc:3037] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.448711 14314 raft_consensus.cc:686] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [term 2 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.448958 14314 consensus_queue.cc:237] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.450002 13246 ts_tablet_manager.cc:1888] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.451838 13120 raft_consensus.cc:2454] T 3bb5a12521044102ae55ca953f1496f9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.452287 12922 leader_election.cc:302] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.452713 14274 raft_consensus.cc:2781] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.453058 14274 raft_consensus.cc:686] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.453457 13246 log.cc:1196] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/472c63d8995346d78633154a8f0d4c2b
I20240506 19:56:52.453366 14274 consensus_queue.cc:237] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.452764 13243 rpcz_store.cc:276] 0506 19:56:51.187213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.192282 (+ 5069us) service_pool.cc:225] Handling call
0506 19:56:52.448049 (+1255767us) ts_tablet_manager.cc:1120] Deleting tablet 2a3291123a1e42af80dbfb578c855618
0506 19:56:52.448059 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.448201 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255570}
I20240506 19:56:52.453799 13246 ts_tablet_manager.cc:1909] T 472c63d8995346d78633154a8f0d4c2b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.454650 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 828) took 2451 ms. Trace:
I20240506 19:56:52.454766 13246 rpcz_store.cc:276] 0506 19:56:50.002736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.004102 (+ 1366us) service_pool.cc:225] Handling call
0506 19:56:52.436356 (+2432254us) ts_tablet_manager.cc:1120] Deleting tablet 472c63d8995346d78633154a8f0d4c2b
0506 19:56:52.436366 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.446284 (+ 9918us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.449909 (+ 3625us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.454611 (+ 4702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432023}
W20240506 19:56:52.454913 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2a3291123a1e42af80dbfb578c855618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a3291123a1e42af80dbfb578c855618 already in progress: creating tablet
I20240506 19:56:52.455256 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 830) took 2449 ms. Trace:
I20240506 19:56:52.455348 13246 rpcz_store.cc:276] 0506 19:56:50.005593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.005694 (+ 101us) service_pool.cc:225] Handling call
0506 19:56:52.455113 (+2449419us) ts_tablet_manager.cc:1120] Deleting tablet 8321457717fd4ad28ae1292d9e9a8669
0506 19:56:52.455126 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.455240 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432832}
I20240506 19:56:52.455726 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 829) took 2450 ms. Trace:
I20240506 19:56:52.455813 13246 rpcz_store.cc:276] 0506 19:56:50.004874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.005214 (+ 340us) service_pool.cc:225] Handling call
0506 19:56:52.455620 (+2450406us) ts_tablet_manager.cc:1120] Deleting tablet 46fd9906504e4b95a5b05b01dc3ea1c0
0506 19:56:52.455630 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.455712 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2433220}
I20240506 19:56:52.456102 13246 tablet_replica.cc:330] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.456355 13246 raft_consensus.cc:2229] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.456518 13246 raft_consensus.cc:2258] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.457145 13246 ts_tablet_manager.cc:1875] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.456506 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1615f6167aae4dc096b9f563ab5e6215 (table default.Table_feb73e349178483fb087f43a269605b0 [id=b19563cb49834a49924bb9d06b60b6ed]) successfully deleted
I20240506 19:56:52.458216 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 913) took 1270 ms. Trace:
I20240506 19:56:52.458962 13243 rpcz_store.cc:276] 0506 19:56:51.187302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.192465 (+ 5163us) service_pool.cc:225] Handling call
0506 19:56:52.456740 (+1264275us) ts_tablet_manager.cc:1120] Deleting tablet 9c93117bdcf44c719c2e3588de538e96
0506 19:56:52.456981 (+ 241us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.457954 (+ 973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264148}
W20240506 19:56:52.461231 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9c93117bdcf44c719c2e3588de538e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c93117bdcf44c719c2e3588de538e96 already in progress: creating tablet
I20240506 19:56:52.460376 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 472c63d8995346d78633154a8f0d4c2b (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
W20240506 19:56:52.464594 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8321457717fd4ad28ae1292d9e9a8669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8321457717fd4ad28ae1292d9e9a8669 already in progress: creating tablet
I20240506 19:56:52.466306 13245 tablet_bootstrap.cc:492] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.466449 13245 ts_tablet_manager.cc:1367] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.140s user 0.013s sys 0.009s
I20240506 19:56:52.467191 13245 raft_consensus.cc:348] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.467401 13245 raft_consensus.cc:374] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.467456 13245 raft_consensus.cc:729] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:52.468233 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46fd9906504e4b95a5b05b01dc3ea1c0 already in progress: creating tablet
I20240506 19:56:52.467698 13245 consensus_queue.cc:260] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.469040 13244 tablet_bootstrap.cc:492] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:52.469664 13245 ts_tablet_manager.cc:1398] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:52.470278 13249 tablet_bootstrap.cc:492] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.470420 13249 ts_tablet_manager.cc:1367] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.004s
I20240506 19:56:52.471107 13249 raft_consensus.cc:348] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.471297 13249 raft_consensus.cc:374] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.471352 13249 raft_consensus.cc:729] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.471160 13244 ts_tablet_manager.cc:1367] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.005s
I20240506 19:56:52.471556 13249 consensus_queue.cc:260] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.471765 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 914) took 1284 ms. Trace:
I20240506 19:56:52.472851 13243 rpcz_store.cc:276] 0506 19:56:51.187405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.192590 (+ 5185us) service_pool.cc:225] Handling call
0506 19:56:52.463808 (+1271218us) ts_tablet_manager.cc:1120] Deleting tablet 5bbcc40911034f319f489931015ca257
0506 19:56:52.471637 (+ 7829us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.471747 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271066}
I20240506 19:56:52.473407 13243 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 915) took 1285 ms. Trace:
I20240506 19:56:52.473562 13245 tablet_bootstrap.cc:492] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.473563 13243 rpcz_store.cc:276] 0506 19:56:51.187524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.192745 (+ 5221us) service_pool.cc:225] Handling call
0506 19:56:52.473224 (+1280479us) ts_tablet_manager.cc:1120] Deleting tablet 2b3268094cbf481698cb373c1bf6dfba
0506 19:56:52.473233 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.473388 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280344}
I20240506 19:56:52.474880 13243 tablet_replica.cc:330] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.475021 13245 tablet_bootstrap.cc:654] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.475575 13243 raft_consensus.cc:2229] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.475903 13243 pending_rounds.cc:62] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:52.476017 13243 pending_rounds.cc:68] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744087273721856 op_type: NO_OP noop_request { }
I20240506 19:56:52.476183 13243 raft_consensus.cc:2866] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:52.476328 13243 raft_consensus.cc:2258] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.472041 13249 ts_tablet_manager.cc:1398] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:52.476920 13249 tablet_bootstrap.cc:492] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.477773 13243 ts_tablet_manager.cc:1875] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:52.478679 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5bbcc40911034f319f489931015ca257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bbcc40911034f319f489931015ca257 already in progress: creating tablet
W20240506 19:56:52.479045 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 691e312510a440898a58bec43ab26e8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691e312510a440898a58bec43ab26e8f already in progress: creating tablet
W20240506 19:56:52.479230 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d43da1d12ce647dcbcd31263932b2f54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43da1d12ce647dcbcd31263932b2f54 already in progress: creating tablet
W20240506 19:56:52.479389 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet af2741336d5a4be1b5306319fc20cdc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af2741336d5a4be1b5306319fc20cdc6 already in progress: creating tablet
W20240506 19:56:52.479563 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet e6d37bb1c2144c4fa71aab6113b9f06e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6d37bb1c2144c4fa71aab6113b9f06e already in progress: creating tablet
W20240506 19:56:52.479722 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c8eeaa71ac61424796c6ffdd993de9db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8eeaa71ac61424796c6ffdd993de9db already in progress: creating tablet
W20240506 19:56:52.479892 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 already in progress: creating tablet
W20240506 19:56:52.480055 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 15ed0c7b4569490b91c54d5bdb187e49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ed0c7b4569490b91c54d5bdb187e49 already in progress: creating tablet
W20240506 19:56:52.480249 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 6a7715dad8184493b085d891428ad44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a7715dad8184493b085d891428ad44c already in progress: creating tablet
I20240506 19:56:52.480662 13249 tablet_bootstrap.cc:654] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.481735 13242 ts_tablet_manager.cc:1888] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.481845 13242 log.cc:1196] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/0114edac976844619e438c17bb959010
I20240506 19:56:52.482116 13242 ts_tablet_manager.cc:1909] T 0114edac976844619e438c17bb959010 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.482419 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 752) took 3375 ms. Trace:
I20240506 19:56:52.482501 13242 rpcz_store.cc:276] 0506 19:56:49.106998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.141656 (+ 34658us) service_pool.cc:225] Handling call
0506 19:56:51.978990 (+2837334us) ts_tablet_manager.cc:1120] Deleting tablet 0114edac976844619e438c17bb959010
0506 19:56:51.979004 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.451728 (+472724us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.481621 (+ 29893us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.482395 (+ 774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2808050}
I20240506 19:56:52.482998 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 0114edac976844619e438c17bb959010 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]) successfully deleted
I20240506 19:56:52.483343 13242 tablet_replica.cc:330] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:52.486310 13243 ts_tablet_manager.cc:1888] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.494935 13243 log.cc:1196] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1a6e0126d9854215a0e174a76c0a0391
I20240506 19:56:52.495869 13246 ts_tablet_manager.cc:1888] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.496470 13243 ts_tablet_manager.cc:1909] T 1a6e0126d9854215a0e174a76c0a0391 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.506536 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:52.509852 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet fdab4110236e4f5c8690f6d963ff27f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.512953 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5c08c51b4a7848aa8b49a78e020d7d6c" } new_table_name: "default.Table_1b37b87a1904426883b754783f6d5c2e" modify_external_catalogs: true
I20240506 19:56:52.513540 13242 raft_consensus.cc:2229] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.513736 13242 raft_consensus.cc:2258] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.514299 13242 ts_tablet_manager.cc:1875] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.516825 13243 tablet_replica.cc:330] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.516928 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1a6e0126d9854215a0e174a76c0a0391 (table default.Table_826f8c31905e4a76b9bd194c717a5853 [id=577dd46fb1b44fd380574a6cd0ba2695]) successfully deleted
I20240506 19:56:52.517077 13246 log.cc:1196] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:52.517524 13246 ts_tablet_manager.cc:1909] T 971db6ac24d54efca95695ed1c90d4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.518534 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 832) took 2470 ms. Trace:
I20240506 19:56:52.518658 13246 rpcz_store.cc:276] 0506 19:56:50.047915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.048289 (+ 374us) service_pool.cc:225] Handling call
0506 19:56:52.456054 (+2407765us) ts_tablet_manager.cc:1120] Deleting tablet 971db6ac24d54efca95695ed1c90d4ed
0506 19:56:52.456064 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.460756 (+ 4692us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.495726 (+ 34970us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.518502 (+ 22776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407546}
I20240506 19:56:52.519089 13246 tablet_replica.cc:330] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.519351 13246 raft_consensus.cc:2229] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.519613 13246 raft_consensus.cc:2258] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.520305 13246 ts_tablet_manager.cc:1875] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:52.522564 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e32a40abc4e74586af79474e4b1675c5 consensus state has no leader
I20240506 19:56:52.523042 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_44b86b8ac10f4b758b4356ccf982efb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.523692 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44b86b8ac10f4b758b4356ccf982efb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.523972 13243 raft_consensus.cc:2229] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.524353 13243 pending_rounds.cc:62] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:52.524407 13243 pending_rounds.cc:68] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744085851119616 op_type: NO_OP noop_request { }
I20240506 19:56:52.524551 13243 raft_consensus.cc:2866] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:52.524645 13243 raft_consensus.cc:2258] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.524941 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4f4213fa718b491881f304b4e8f0bf7a
I20240506 19:56:52.525806 13243 ts_tablet_manager.cc:1875] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.525727 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6c36d4fe94e7454280ad13d12a5ed6db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.526239 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c36d4fe94e7454280ad13d12a5ed6db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.528309 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_330c25efd3544e359efe8b4e731ce71c" } modify_external_catalogs: true
I20240506 19:56:52.528468 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_330c25efd3544e359efe8b4e731ce71c" } modify_external_catalogs: true
I20240506 19:56:52.529695 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f424e713b96d43cdac6597c20bd5fc73
I20240506 19:56:52.530382 13249 tablet_bootstrap.cc:492] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.530508 13249 ts_tablet_manager.cc:1367] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.054s user 0.015s sys 0.030s
I20240506 19:56:52.530449 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_540b5b87de2e447a8ce8a0b8f758cab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.530944 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_540b5b87de2e447a8ce8a0b8f758cab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.531273 13249 raft_consensus.cc:348] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.531442 13249 raft_consensus.cc:374] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.531492 13249 raft_consensus.cc:729] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.531725 13249 consensus_queue.cc:260] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:52.532163 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 196b339d0bc54426b69a60fe59f1af00 consensus state has no leader
I20240506 19:56:52.532548 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_153effa513de4385b0ddfd1d561446ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.532977 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_153effa513de4385b0ddfd1d561446ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.534214 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 979d789bc89a435b9e6cd8d6dc9c2037 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:52.534826 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_62d8dd9a1336466d8d008ec2ca4ff8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.535245 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d8dd9a1336466d8d008ec2ca4ff8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.536882 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea76ff9f6fb54cac83b450414487d2d3" } new_table_name: "default.Table_28ea2a7143b84c1c9eeb79da8bf201ad" modify_external_catalogs: true
I20240506 19:56:52.537967 12513 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet e51825f2e6b546309abb3a2bf41322aa
I20240506 19:56:52.538091 13249 ts_tablet_manager.cc:1398] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:56:52.538151 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet fdab4110236e4f5c8690f6d963ff27f7 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:52.538524 12513 master_service.cc:946] ReplaceTablet: received request to replace tablet 260d7ab458794df496012cb80466f59c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:52.539491 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4213fa718b491881f304b4e8f0bf7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.539676 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet f424e713b96d43cdac6597c20bd5fc73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.540709 13246 ts_tablet_manager.cc:1888] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240506 19:56:52.540839 13246 log.cc:1196] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/41faccd54f3e49ddb48ea5234f6cdab5
I20240506 19:56:52.541169 13246 ts_tablet_manager.cc:1909] T 41faccd54f3e49ddb48ea5234f6cdab5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.541978 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 831) took 2494 ms. Trace:
I20240506 19:56:52.542068 13246 rpcz_store.cc:276] 0506 19:56:50.047790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.077200 (+ 29410us) service_pool.cc:225] Handling call
0506 19:56:52.519028 (+2441828us) ts_tablet_manager.cc:1120] Deleting tablet 41faccd54f3e49ddb48ea5234f6cdab5
0506 19:56:52.519042 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.537769 (+ 18727us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.540609 (+ 2840us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.541954 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2441555}
I20240506 19:56:52.542495 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 833) took 2424 ms. Trace:
I20240506 19:56:52.542563 13246 rpcz_store.cc:276] 0506 19:56:50.118125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.119056 (+ 931us) service_pool.cc:225] Handling call
0506 19:56:52.542364 (+2423308us) ts_tablet_manager.cc:1120] Deleting tablet f078559df48f4d23869ddcb4b3fca9fc
0506 19:56:52.542375 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.542477 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2423035}
I20240506 19:56:52.542793 13246 tablet_replica.cc:330] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.542958 13246 raft_consensus.cc:2229] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.543099 13246 raft_consensus.cc:2258] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.543598 13246 ts_tablet_manager.cc:1875] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.544096 13249 tablet_bootstrap.cc:492] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.545624 13249 tablet_bootstrap.cc:654] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.546545 14211 raft_consensus.cc:482] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.546700 14211 raft_consensus.cc:504] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.547387 14211 leader_election.cc:288] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.547801 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4213fa718b491881f304b4e8f0bf7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.547873 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04738def56324a368f8182543107e32e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.548027 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.548002 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04738def56324a368f8182543107e32e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.548173 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet f424e713b96d43cdac6597c20bd5fc73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.548205 13243 ts_tablet_manager.cc:1888] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.548314 13243 log.cc:1196] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/4acde41b2c03440a89243aae008779e1
W20240506 19:56:52.548413 13057 leader_election.cc:341] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:52.548549 13058 leader_election.cc:341] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.548647 13058 leader_election.cc:302] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.549041 14211 raft_consensus.cc:2726] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.553824 13243 ts_tablet_manager.cc:1909] T 4acde41b2c03440a89243aae008779e1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.555374 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 4acde41b2c03440a89243aae008779e1 (table default.Table_fe3e87d2202147518431c995667b07fc [id=b7788b3408254775a7b73c81911108c2]) successfully deleted
I20240506 19:56:52.555982 13249 tablet_bootstrap.cc:492] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.556128 13249 ts_tablet_manager.cc:1367] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
W20240506 19:56:52.556347 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 44ce8749846847108edf5f554139cbd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ce8749846847108edf5f554139cbd9 already in progress: creating tablet
I20240506 19:56:52.556723 13243 tablet_replica.cc:330] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.556967 13243 raft_consensus.cc:2229] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:52.556926 13249 raft_consensus.cc:348] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.557159 13249 raft_consensus.cc:374] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.557241 13249 raft_consensus.cc:729] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.557494 13249 consensus_queue.cc:260] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.557504 13243 raft_consensus.cc:2258] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.558442 13249 ts_tablet_manager.cc:1398] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:52.558741 13249 tablet_bootstrap.cc:492] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.517261 13244 raft_consensus.cc:348] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.559957 13244 raft_consensus.cc:374] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.560544 13249 tablet_bootstrap.cc:654] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.560497 13244 raft_consensus.cc:729] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.561404 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4213fa718b491881f304b4e8f0bf7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.562218 13243 ts_tablet_manager.cc:1875] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.566074 14314 raft_consensus.cc:482] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.566211 14314 raft_consensus.cc:504] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.566812 14314 leader_election.cc:288] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.567420 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.568763 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.568967 13120 raft_consensus.cc:2454] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:52.569983 12921 leader_election.cc:341] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.570585 12922 leader_election.cc:302] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.570853 14314 raft_consensus.cc:2781] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.570936 14314 raft_consensus.cc:482] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.570989 14314 raft_consensus.cc:3037] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.573766 14314 raft_consensus.cc:504] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.575361 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.576529 13244 consensus_queue.cc:260] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.579226 13242 ts_tablet_manager.cc:1888] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.579295 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet f424e713b96d43cdac6597c20bd5fc73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.579352 13242 log.cc:1196] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/f0a8b12f36e74c95ad6eaf30e432f3e9
I20240506 19:56:52.579689 13242 ts_tablet_manager.cc:1909] T f0a8b12f36e74c95ad6eaf30e432f3e9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:52.579839 12921 leader_election.cc:341] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.580029 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 755) took 3441 ms. Trace:
I20240506 19:56:52.580744 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.580952 13123 raft_consensus.cc:3037] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.581073 13249 tablet_bootstrap.cc:492] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.581193 13249 ts_tablet_manager.cc:1367] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.001s
I20240506 19:56:52.582016 13249 raft_consensus.cc:348] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.582262 13249 raft_consensus.cc:374] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.582356 13249 raft_consensus.cc:729] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.582584 13249 consensus_queue.cc:260] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.583796 13123 raft_consensus.cc:2454] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.583976 13244 ts_tablet_manager.cc:1398] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.112s user 0.000s sys 0.003s
I20240506 19:56:52.584522 12922 leader_election.cc:302] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.585558 14274 raft_consensus.cc:2781] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.585675 14274 raft_consensus.cc:686] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.585889 14274 consensus_queue.cc:237] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.586385 13249 ts_tablet_manager.cc:1398] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:52.586639 13249 tablet_bootstrap.cc:492] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.588999 13249 tablet_bootstrap.cc:654] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.590041 14314 leader_election.cc:288] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.580118 13242 rpcz_store.cc:276] 0506 19:56:49.138741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.146311 (+ 7570us) service_pool.cc:225] Handling call
0506 19:56:52.483292 (+3336981us) ts_tablet_manager.cc:1120] Deleting tablet f0a8b12f36e74c95ad6eaf30e432f3e9
0506 19:56:52.483303 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.549744 (+ 66441us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.574762 (+ 25018us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.580007 (+ 5245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3311821}
I20240506 19:56:52.597671 13242 tablet_replica.cc:330] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:52.597819 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet f0a8b12f36e74c95ad6eaf30e432f3e9 (table default.Table_038664bd22da4e20b40b21843516298f [id=34aeae2e9b5546d1b405ce1432e3c0cb]) successfully deleted
I20240506 19:56:52.597908 13242 raft_consensus.cc:2229] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.598073 13242 raft_consensus.cc:2258] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.598659 13242 ts_tablet_manager.cc:1875] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.599119 13245 tablet_bootstrap.cc:492] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.599316 13245 ts_tablet_manager.cc:1367] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.126s user 0.000s sys 0.011s
I20240506 19:56:52.600200 13245 raft_consensus.cc:348] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.600468 13245 raft_consensus.cc:374] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.600723 13245 raft_consensus.cc:729] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.601001 13245 consensus_queue.cc:260] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.601971 13249 tablet_bootstrap.cc:492] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.602150 13249 ts_tablet_manager.cc:1367] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240506 19:56:52.602959 13249 raft_consensus.cc:348] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.603210 13249 raft_consensus.cc:374] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.603315 13249 raft_consensus.cc:729] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.603590 13249 consensus_queue.cc:260] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.604158 13249 ts_tablet_manager.cc:1398] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:52.604563 13249 tablet_bootstrap.cc:492] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.606235 13249 tablet_bootstrap.cc:654] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.606645 13245 ts_tablet_manager.cc:1398] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240506 19:56:52.606935 13245 tablet_bootstrap.cc:492] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.607815 13243 ts_tablet_manager.cc:1888] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.609244 13246 ts_tablet_manager.cc:1888] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.615517 13246 log.cc:1196] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/4acde41b2c03440a89243aae008779e1
I20240506 19:56:52.616057 13246 ts_tablet_manager.cc:1909] T 4acde41b2c03440a89243aae008779e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.616245 13243 log.cc:1196] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/67e8b8be953a4a5784f2d098fd6364e3
I20240506 19:56:52.616776 13243 ts_tablet_manager.cc:1909] T 67e8b8be953a4a5784f2d098fd6364e3 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.617108 13244 tablet_bootstrap.cc:492] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:52.613713 13245 tablet_bootstrap.cc:654] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.621419 13249 tablet_bootstrap.cc:492] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.621613 13249 ts_tablet_manager.cc:1367] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.001s
I20240506 19:56:52.622180 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 67e8b8be953a4a5784f2d098fd6364e3 (table default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb [id=7a2c513cd8e74817b4a391c9c7932085]) successfully deleted
I20240506 19:56:52.622519 13249 raft_consensus.cc:348] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.622797 13249 raft_consensus.cc:374] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.622812 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 835) took 2380 ms. Trace:
I20240506 19:56:52.622946 13249 raft_consensus.cc:729] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.622896 13246 rpcz_store.cc:276] 0506 19:56:50.242623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.242685 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:52.542755 (+2300070us) ts_tablet_manager.cc:1120] Deleting tablet 4acde41b2c03440a89243aae008779e1
0506 19:56:52.542763 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.595096 (+ 52333us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.609149 (+ 14053us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.622785 (+ 13636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2299863}
I20240506 19:56:52.623267 13246 tablet_replica.cc:330] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.623360 13249 consensus_queue.cc:260] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.624073 13249 ts_tablet_manager.cc:1398] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:52.624157 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 971db6ac24d54efca95695ed1c90d4ed (table default.Table_cc8e3aa90fea4998b3e0a4a411add544 [id=86b683f4dec545118e95b30b194e1fa5]) successfully deleted
W20240506 19:56:52.624153 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8a9b92795a7742caa2ee99ab7ca138bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a9b92795a7742caa2ee99ab7ca138bf already in progress: creating tablet
I20240506 19:56:52.624411 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]) successfully deleted
I20240506 19:56:52.624536 13249 tablet_bootstrap.cc:492] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:52.624622 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f078559df48f4d23869ddcb4b3fca9fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f078559df48f4d23869ddcb4b3fca9fc already in progress: creating tablet
I20240506 19:56:52.624812 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 4acde41b2c03440a89243aae008779e1 (table default.Table_fe3e87d2202147518431c995667b07fc [id=b7788b3408254775a7b73c81911108c2]) successfully deleted
W20240506 19:56:52.624843 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 35198d5786c14de89c9751a0372ba91d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35198d5786c14de89c9751a0372ba91d already in progress: creating tablet
I20240506 19:56:52.623461 13246 raft_consensus.cc:2229] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.625257 13246 raft_consensus.cc:2258] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:52.625869 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 910d1465df784dfba6d78ff7fd9f2722 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 910d1465df784dfba6d78ff7fd9f2722 already in progress: creating tablet
I20240506 19:56:52.626328 13249 tablet_bootstrap.cc:654] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.626585 13246 ts_tablet_manager.cc:1875] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.628931 14211 raft_consensus.cc:482] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.629055 14211 raft_consensus.cc:504] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.629699 14211 leader_election.cc:288] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:52.629802 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5fe8d4891c484627ac729971ff7aad3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fe8d4891c484627ac729971ff7aad3e already in progress: creating tablet
I20240506 19:56:52.630218 13242 ts_tablet_manager.cc:1888] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.630365 13242 log.cc:1196] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d05c5ca7ff864fa98f3494032729e978
I20240506 19:56:52.630389 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973e600038742a6bec0bb2d8f3551fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.630422 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973e600038742a6bec0bb2d8f3551fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.630702 13242 ts_tablet_manager.cc:1909] T d05c5ca7ff864fa98f3494032729e978 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:52.630939 13058 leader_election.cc:341] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.631037 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 756) took 3449 ms. Trace:
I20240506 19:56:52.631116 13242 rpcz_store.cc:276] 0506 19:56:49.181366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.185086 (+ 3720us) service_pool.cc:225] Handling call
0506 19:56:52.597613 (+3412527us) ts_tablet_manager.cc:1120] Deleting tablet d05c5ca7ff864fa98f3494032729e978
0506 19:56:52.597624 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.615659 (+ 18035us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.621219 (+ 5560us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.631018 (+ 9799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3412191}
W20240506 19:56:52.631405 13057 leader_election.cc:341] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.631539 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 751) took 3540 ms. Trace:
I20240506 19:56:52.631539 13057 leader_election.cc:302] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.631615 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d05c5ca7ff864fa98f3494032729e978 (table default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]) successfully deleted
I20240506 19:56:52.631619 13242 rpcz_store.cc:276] 0506 19:56:49.091102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.192262 (+101160us) service_pool.cc:225] Handling call
0506 19:56:52.631425 (+3439163us) ts_tablet_manager.cc:1120] Deleting tablet fdef0a40f2e346a798b01e83a960de09
0506 19:56:52.631436 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.631522 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3438934}
I20240506 19:56:52.631841 14211 raft_consensus.cc:2726] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.631949 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 757) took 3311 ms. Trace:
I20240506 19:56:52.632018 13242 rpcz_store.cc:276] 0506 19:56:49.320342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.322332 (+ 1990us) service_pool.cc:225] Handling call
0506 19:56:52.631846 (+3309514us) ts_tablet_manager.cc:1120] Deleting tablet f63e8e12fd3b453a96413f1fc769fb9f
0506 19:56:52.631857 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.631933 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309181}
I20240506 19:56:52.632261 13242 tablet_replica.cc:330] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:52.632258 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet fdef0a40f2e346a798b01e83a960de09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdef0a40f2e346a798b01e83a960de09 already in progress: creating tablet
I20240506 19:56:52.632440 13242 raft_consensus.cc:2229] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.632593 13242 raft_consensus.cc:2258] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:52.632655 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f63e8e12fd3b453a96413f1fc769fb9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63e8e12fd3b453a96413f1fc769fb9f already in progress: creating tablet
I20240506 19:56:52.633175 13242 ts_tablet_manager.cc:1875] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:52.633800 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4f4213fa718b491881f304b4e8f0bf7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f4213fa718b491881f304b4e8f0bf7a already in progress: creating tablet
I20240506 19:56:52.634094 13243 tablet_replica.cc:330] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:52.634313 13243 raft_consensus.cc:2229] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.634488 13243 raft_consensus.cc:2258] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.634994 13243 ts_tablet_manager.cc:1875] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.635593 13244 tablet_bootstrap.cc:654] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.638069 13245 tablet_bootstrap.cc:492] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.638532 13245 ts_tablet_manager.cc:1367] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20240506 19:56:52.639462 13245 raft_consensus.cc:348] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.639720 13245 raft_consensus.cc:374] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.639823 13245 raft_consensus.cc:729] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.639871 13249 tablet_bootstrap.cc:492] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.640070 13249 ts_tablet_manager.cc:1367] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20240506 19:56:52.640105 13245 consensus_queue.cc:260] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.640784 13245 ts_tablet_manager.cc:1398] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:52.640879 13249 raft_consensus.cc:348] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.641098 13245 tablet_bootstrap.cc:492] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.641103 13249 raft_consensus.cc:374] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.641247 13249 raft_consensus.cc:729] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.641530 13249 consensus_queue.cc:260] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.642197 13249 ts_tablet_manager.cc:1398] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:52.642570 13249 tablet_bootstrap.cc:492] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.643294 13245 tablet_bootstrap.cc:654] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.644172 13249 tablet_bootstrap.cc:654] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.654891 13246 ts_tablet_manager.cc:1888] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.655021 13246 log.cc:1196] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:52.655395 13246 ts_tablet_manager.cc:1909] T 6b59ad067c274fddbba5f71e89480dec P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.656266 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 836) took 2264 ms. Trace:
I20240506 19:56:52.656365 13246 rpcz_store.cc:276] 0506 19:56:50.391393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.391471 (+ 78us) service_pool.cc:225] Handling call
0506 19:56:52.623208 (+2231737us) ts_tablet_manager.cc:1120] Deleting tablet 6b59ad067c274fddbba5f71e89480dec
0506 19:56:52.623218 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.651151 (+ 27933us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.654787 (+ 3636us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.656242 (+ 1455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231471}
I20240506 19:56:52.656752 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 853) took 2093 ms. Trace:
I20240506 19:56:52.656829 13246 rpcz_store.cc:276] 0506 19:56:50.563445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.563948 (+ 503us) service_pool.cc:225] Handling call
0506 19:56:52.656637 (+2092689us) ts_tablet_manager.cc:1120] Deleting tablet 110a046c2db14758b8d28616ae63c18c
0506 19:56:52.656647 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.656739 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2092503}
I20240506 19:56:52.657624 13249 tablet_bootstrap.cc:492] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.657842 13249 ts_tablet_manager.cc:1367] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240506 19:56:52.658627 14274 raft_consensus.cc:482] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.658753 14274 raft_consensus.cc:504] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.659401 14274 leader_election.cc:288] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.659960 13249 raft_consensus.cc:348] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.660002 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.660210 13249 raft_consensus.cc:374] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.660387 13249 raft_consensus.cc:729] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.660626 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.660630 13249 consensus_queue.cc:260] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.660820 13123 raft_consensus.cc:2454] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.661446 13242 ts_tablet_manager.cc:1888] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.661578 13242 log.cc:1196] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/aae313961a914ed485f866d85785a788
W20240506 19:56:52.661621 12921 leader_election.cc:341] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.661898 13242 ts_tablet_manager.cc:1909] T aae313961a914ed485f866d85785a788 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.661971 12922 leader_election.cc:302] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.662211 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 758) took 3271 ms. Trace:
I20240506 19:56:52.662405 14274 raft_consensus.cc:2781] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.662292 13242 rpcz_store.cc:276] 0506 19:56:49.390655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.390712 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:52.632214 (+3241502us) ts_tablet_manager.cc:1120] Deleting tablet aae313961a914ed485f866d85785a788
0506 19:56:52.632223 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.645767 (+ 13544us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.658789 (+ 13022us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.662190 (+ 3401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3241280}
I20240506 19:56:52.662494 14274 raft_consensus.cc:482] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.662554 14274 raft_consensus.cc:3037] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.662616 13242 tablet_replica.cc:330] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:52.662812 13242 raft_consensus.cc:2229] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.662971 13242 raft_consensus.cc:2258] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.663408 13249 ts_tablet_manager.cc:1398] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240506 19:56:52.663492 13242 ts_tablet_manager.cc:1875] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.663652 13249 tablet_bootstrap.cc:492] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:52.664688 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2be78cbc8264f6e8d2e05137135161c consensus state has no leader
I20240506 19:56:52.665122 13249 tablet_bootstrap.cc:654] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.665017 14274 raft_consensus.cc:504] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.666224 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:52.666427 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:52.666906 12921 leader_election.cc:341] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.667559 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.667732 13123 raft_consensus.cc:3037] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.668682 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 8321457717fd4ad28ae1292d9e9a8669 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:52.669225 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5a100640805e4e989b84032702a4de27" } new_table_name: "default.Table_9068971de53c4d30bbe309692dd907f3" modify_external_catalogs: true
I20240506 19:56:52.670429 13123 raft_consensus.cc:2454] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.670874 12922 leader_election.cc:302] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.670944 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b35d5d1b38604d24bf72370ec6d5998c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:52.671144 14314 raft_consensus.cc:2781] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.671248 14314 raft_consensus.cc:686] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
W20240506 19:56:52.671458 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b35d5d1b38604d24bf72370ec6d5998c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.671476 14314 consensus_queue.cc:237] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.672591 14274 leader_election.cc:288] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.673136 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e6b24176d9d941d4a9dfb97e24b91ab6
I20240506 19:56:52.673905 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6f73fec327d24cf7bfabb15d38171100" } modify_external_catalogs: true
I20240506 19:56:52.674083 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6f73fec327d24cf7bfabb15d38171100" } modify_external_catalogs: true
I20240506 19:56:52.674685 13243 ts_tablet_manager.cc:1888] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.674777 13243 log.cc:1196] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/bd2c81ac92d04ddc93cc52e5c01cbc60
I20240506 19:56:52.674866 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7cbfc903cd8e400e87d5f107413810f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:52.675072 13243 ts_tablet_manager.cc:1909] T bd2c81ac92d04ddc93cc52e5c01cbc60 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:52.675079 13249 tablet_bootstrap.cc:492] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.675252 13249 ts_tablet_manager.cc:1367] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
W20240506 19:56:52.675374 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cbfc903cd8e400e87d5f107413810f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.676095 13249 raft_consensus.cc:348] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.676342 13249 raft_consensus.cc:374] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.676421 13249 raft_consensus.cc:729] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:52.676774 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a6db7d0ad024f53ad76920bcbc0248e consensus state has no leader
I20240506 19:56:52.676779 13249 consensus_queue.cc:260] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.676900 13242 ts_tablet_manager.cc:1888] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.677008 13242 log.cc:1196] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/4be929f0e72e4d22a9af2fe6b3072bbe
I20240506 19:56:52.677325 13242 ts_tablet_manager.cc:1909] T 4be929f0e72e4d22a9af2fe6b3072bbe P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.677424 13245 tablet_bootstrap.cc:492] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.677479 13249 ts_tablet_manager.cc:1398] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:52.677603 13245 ts_tablet_manager.cc:1367] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240506 19:56:52.677624 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 759) took 3273 ms. Trace:
I20240506 19:56:52.677734 13249 tablet_bootstrap.cc:492] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.677698 13242 rpcz_store.cc:276] 0506 19:56:49.403807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.403884 (+ 77us) service_pool.cc:225] Handling call
0506 19:56:52.662570 (+3258686us) ts_tablet_manager.cc:1120] Deleting tablet 4be929f0e72e4d22a9af2fe6b3072bbe
0506 19:56:52.662579 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.673928 (+ 11349us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.676810 (+ 2882us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.677603 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3183801}
I20240506 19:56:52.677930 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet bd2c81ac92d04ddc93cc52e5c01cbc60 (table default.Table_ca6325447fde4d198454d0ac20e928bd [id=a5b2fb638e5a4ee2abfd610cb6274767]) successfully deleted
I20240506 19:56:52.678045 13242 tablet_replica.cc:330] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:52.678215 13242 raft_consensus.cc:2229] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.678368 13242 raft_consensus.cc:2258] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.678459 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b24176d9d941d4a9dfb97e24b91ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:52.678290 13245 raft_consensus.cc:348] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.678755 13245 raft_consensus.cc:374] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.678809 13245 raft_consensus.cc:729] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.678918 13242 ts_tablet_manager.cc:1875] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.679025 13245 consensus_queue.cc:260] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.679268 13249 tablet_bootstrap.cc:654] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.679683 13245 ts_tablet_manager.cc:1398] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:52.679946 13245 tablet_bootstrap.cc:492] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.681458 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b24176d9d941d4a9dfb97e24b91ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.681676 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet aae313961a914ed485f866d85785a788 (table default.Table_632142c0692a45b69cba3c19adb0b444 [id=a63e1c9e058a4501831a67516affe578]) successfully deleted
I20240506 19:56:52.681911 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 4be929f0e72e4d22a9af2fe6b3072bbe (table default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]) successfully deleted
W20240506 19:56:52.681967 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f424e713b96d43cdac6597c20bd5fc73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f424e713b96d43cdac6597c20bd5fc73 already in progress: creating tablet
W20240506 19:56:52.686947 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet e6b24176d9d941d4a9dfb97e24b91ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6b24176d9d941d4a9dfb97e24b91ab6 already in progress: creating tablet
I20240506 19:56:52.687698 13245 tablet_bootstrap.cc:654] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.688975 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 855) took 2120 ms. Trace:
I20240506 19:56:52.689306 13246 rpcz_store.cc:276] 0506 19:56:50.567188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.567244 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:52.657061 (+2089817us) ts_tablet_manager.cc:1120] Deleting tablet 0bc56299ed854d6c8ef201127692fd60
0506 19:56:52.657068 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.687871 (+ 30803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2089624}
I20240506 19:56:52.690649 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 856) took 2122 ms. Trace:
I20240506 19:56:52.690798 13246 rpcz_store.cc:276] 0506 19:56:50.568108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.568157 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:52.690537 (+2122380us) ts_tablet_manager.cc:1120] Deleting tablet 4c2eb5dac9d34fb09119868bf5a85803
0506 19:56:52.690547 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.690634 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2122212}
I20240506 19:56:52.691246 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 854) took 2125 ms. Trace:
I20240506 19:56:52.691356 13246 rpcz_store.cc:276] 0506 19:56:50.566078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.568741 (+ 2663us) service_pool.cc:225] Handling call
0506 19:56:52.691108 (+2122367us) ts_tablet_manager.cc:1120] Deleting tablet 269210fbc3d4446181b123f200aae472
0506 19:56:52.691116 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.691232 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2122206}
I20240506 19:56:52.692588 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 857) took 2119 ms. Trace:
I20240506 19:56:52.692708 13246 rpcz_store.cc:276] 0506 19:56:50.573064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.573621 (+ 557us) service_pool.cc:225] Handling call
0506 19:56:52.692497 (+2118876us) ts_tablet_manager.cc:1120] Deleting tablet 1e64cb2f31b64b07848c2dfed18ab9f1
0506 19:56:52.692506 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.692574 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2118696}
I20240506 19:56:52.693039 13246 tablet_replica.cc:330] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.694456 13246 raft_consensus.cc:2229] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.694708 13246 raft_consensus.cc:2258] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.695307 13246 ts_tablet_manager.cc:1875] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.695384 13249 tablet_bootstrap.cc:492] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.695557 13249 ts_tablet_manager.cc:1367] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240506 19:56:52.696331 13249 raft_consensus.cc:348] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.696590 13249 raft_consensus.cc:374] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.696679 13249 raft_consensus.cc:729] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.696942 13249 consensus_queue.cc:260] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.699200 13249 ts_tablet_manager.cc:1398] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240506 19:56:52.699532 13249 tablet_bootstrap.cc:492] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.701026 14314 raft_consensus.cc:482] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.701193 13249 tablet_bootstrap.cc:654] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.701153 14314 raft_consensus.cc:504] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.701787 14314 leader_election.cc:288] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:52.702600 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796beb8e0a2c41d698bd4c05796263f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.703604 12921 leader_election.cc:341] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.703593 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796beb8e0a2c41d698bd4c05796263f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.703685 13245 tablet_bootstrap.cc:492] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.703804 13245 ts_tablet_manager.cc:1367] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240506 19:56:52.703781 13123 raft_consensus.cc:2454] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.704195 12922 leader_election.cc:302] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.704711 14314 raft_consensus.cc:2781] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.704787 14314 raft_consensus.cc:482] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.704844 14314 raft_consensus.cc:3037] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.706703 13245 raft_consensus.cc:348] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.706918 13245 raft_consensus.cc:374] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.707012 13245 raft_consensus.cc:729] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.707064 13242 ts_tablet_manager.cc:1888] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:52.707185 13242 log.cc:1196] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/f4e4be565e2d4c59b3587baed72e4a9c
I20240506 19:56:52.707265 13245 consensus_queue.cc:260] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.707284 14314 raft_consensus.cc:504] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.707520 13242 ts_tablet_manager.cc:1909] T f4e4be565e2d4c59b3587baed72e4a9c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.707849 13245 ts_tablet_manager.cc:1398] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:52.707835 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 760) took 3280 ms. Trace:
I20240506 19:56:52.707916 13242 rpcz_store.cc:276] 0506 19:56:49.427346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.427403 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:52.678000 (+3250597us) ts_tablet_manager.cc:1120] Deleting tablet f4e4be565e2d4c59b3587baed72e4a9c
0506 19:56:52.678009 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.702081 (+ 24072us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.706952 (+ 4871us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.707813 (+ 861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198710}
I20240506 19:56:52.708120 13245 tablet_bootstrap.cc:492] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.708340 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796beb8e0a2c41d698bd4c05796263f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.709242 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet f4e4be565e2d4c59b3587baed72e4a9c (table default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]) successfully deleted
I20240506 19:56:52.709672 13245 tablet_bootstrap.cc:654] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:52.709623 12921 leader_election.cc:341] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.710312 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796beb8e0a2c41d698bd4c05796263f1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.710543 13123 raft_consensus.cc:3037] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.713058 13123 raft_consensus.cc:2454] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.713343 13246 ts_tablet_manager.cc:1888] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.713472 13246 log.cc:1196] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:56:52.713860 14314 leader_election.cc:288] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:52.714037 13246 ts_tablet_manager.cc:1909] T bd26e6b8115644cabe6532c7be1bbe95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.714157 12922 leader_election.cc:302] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.714787 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 863) took 2091 ms. Trace:
I20240506 19:56:52.714891 13246 rpcz_store.cc:276] 0506 19:56:50.623638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.685561 (+ 61923us) service_pool.cc:225] Handling call
0506 19:56:52.693001 (+2007440us) ts_tablet_manager.cc:1120] Deleting tablet bd26e6b8115644cabe6532c7be1bbe95
0506 19:56:52.693008 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.702111 (+ 9103us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.713227 (+ 11116us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.714722 (+ 1495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007240}
I20240506 19:56:52.715451 13246 tablet_replica.cc:330] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.715700 13246 raft_consensus.cc:2229] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.715893 13246 raft_consensus.cc:2258] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.716367 14314 raft_consensus.cc:2781] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.716531 14314 raft_consensus.cc:686] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.716804 14314 consensus_queue.cc:237] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.708340 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 762) took 3183 ms. Trace:
I20240506 19:56:52.717617 13246 ts_tablet_manager.cc:1875] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.718428 13242 rpcz_store.cc:276] 0506 19:56:49.524864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.525269 (+ 405us) service_pool.cc:225] Handling call
0506 19:56:52.708227 (+3182958us) ts_tablet_manager.cc:1120] Deleting tablet 3fb398132c58489fb0a28492302e4a23
0506 19:56:52.708237 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.708326 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3182710}
I20240506 19:56:52.718924 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 763) took 3187 ms. Trace:
W20240506 19:56:52.719103 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 3fb398132c58489fb0a28492302e4a23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb398132c58489fb0a28492302e4a23 already in progress: creating tablet
I20240506 19:56:52.719002 13242 rpcz_store.cc:276] 0506 19:56:49.531513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.533446 (+ 1933us) service_pool.cc:225] Handling call
0506 19:56:52.718787 (+3185341us) ts_tablet_manager.cc:1120] Deleting tablet 58c88f80f5b34672b7c3a4b604469bf1
0506 19:56:52.718798 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.718906 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185093}
I20240506 19:56:52.720386 13242 tablet_replica.cc:330] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:52.720554 13242 raft_consensus.cc:2229] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:52.720606 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 58c88f80f5b34672b7c3a4b604469bf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58c88f80f5b34672b7c3a4b604469bf1 already in progress: creating tablet
I20240506 19:56:52.720718 13242 raft_consensus.cc:2258] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.721199 13242 ts_tablet_manager.cc:1875] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.723600 13245 tablet_bootstrap.cc:492] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.726089 13249 tablet_bootstrap.cc:492] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.726229 13249 ts_tablet_manager.cc:1367] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
W20240506 19:56:52.726838 12508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.253:36819, user_credentials={real_user=slave}} blocked reactor thread for 67055.3us
I20240506 19:56:52.726899 13249 raft_consensus.cc:348] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.727044 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 6b59ad067c274fddbba5f71e89480dec (table default.Table_37e9aef6398b47059569bab0a7265a94 [id=d8c76c1f826d49c4818b1e278d407b0e]) successfully deleted
I20240506 19:56:52.727078 13249 raft_consensus.cc:374] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.727120 13249 raft_consensus.cc:729] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:52.727277 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 110a046c2db14758b8d28616ae63c18c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a046c2db14758b8d28616ae63c18c already in progress: creating tablet
I20240506 19:56:52.727296 13249 consensus_queue.cc:260] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:52.727483 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0bc56299ed854d6c8ef201127692fd60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bc56299ed854d6c8ef201127692fd60 already in progress: creating tablet
W20240506 19:56:52.727653 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4c2eb5dac9d34fb09119868bf5a85803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c2eb5dac9d34fb09119868bf5a85803 already in progress: creating tablet
I20240506 19:56:52.727720 13249 ts_tablet_manager.cc:1398] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240506 19:56:52.727790 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 269210fbc3d4446181b123f200aae472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 269210fbc3d4446181b123f200aae472 already in progress: creating tablet
I20240506 19:56:52.727952 13249 tablet_bootstrap.cc:492] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:52.727950 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e64cb2f31b64b07848c2dfed18ab9f1 already in progress: creating tablet
I20240506 19:56:52.728075 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet bd26e6b8115644cabe6532c7be1bbe95 (table default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5 [id=16258fb4a39c4effb6effa8e857b0b6f]) successfully deleted
I20240506 19:56:52.729957 13249 tablet_bootstrap.cc:654] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.732869 14274 raft_consensus.cc:482] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.732996 14274 raft_consensus.cc:504] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.735131 14274 leader_election.cc:288] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:52.735284 13246 ts_tablet_manager.cc:1888] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.735402 13246 log.cc:1196] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/796beb8e0a2c41d698bd4c05796263f1
I20240506 19:56:52.735699 13246 ts_tablet_manager.cc:1909] T 796beb8e0a2c41d698bd4c05796263f1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.735719 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c96f014293242be9640515697c0733b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.735745 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c96f014293242be9640515697c0733b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.735960 13123 raft_consensus.cc:2454] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.736289 12922 leader_election.cc:302] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:52.736425 12921 leader_election.cc:341] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.736537 14274 raft_consensus.cc:2781] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.736619 14274 raft_consensus.cc:482] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.736694 14274 raft_consensus.cc:3037] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.739946 14274 raft_consensus.cc:504] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.742342 14274 leader_election.cc:288] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:52.743167 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c96f014293242be9640515697c0733b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.743659 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c96f014293242be9640515697c0733b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:52.743767 12921 leader_election.cc:341] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.743846 13123 raft_consensus.cc:3037] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.746490 13123 raft_consensus.cc:2454] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.747121 12922 leader_election.cc:302] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.747376 14314 raft_consensus.cc:482] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.747502 14314 raft_consensus.cc:504] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.747622 14274 raft_consensus.cc:2781] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.747810 14274 raft_consensus.cc:686] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.748044 14274 consensus_queue.cc:237] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.748160 14314 leader_election.cc:288] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.748766 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f4861a01404663870ce91978c8af4f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.749249 12921 leader_election.cc:341] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.750020 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f4861a01404663870ce91978c8af4f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.750257 13123 raft_consensus.cc:2454] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.750638 12922 leader_election.cc:302] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.751214 13249 tablet_bootstrap.cc:492] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.751689 14314 raft_consensus.cc:2781] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.751747 13249 ts_tablet_manager.cc:1367] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20240506 19:56:52.751767 14314 raft_consensus.cc:482] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.751829 14314 raft_consensus.cc:3037] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.752453 13249 raft_consensus.cc:348] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.752610 13249 raft_consensus.cc:374] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.752658 13249 raft_consensus.cc:729] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.752847 13249 consensus_queue.cc:260] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.753346 13249 ts_tablet_manager.cc:1398] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:52.753588 13249 tablet_bootstrap.cc:492] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.754459 14314 raft_consensus.cc:504] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.755126 13249 tablet_bootstrap.cc:654] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.755379 13242 ts_tablet_manager.cc:1888] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.755491 13242 log.cc:1196] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/6d04a1c5ee7b4d529fa2f4da4945ff09
I20240506 19:56:52.755472 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f4861a01404663870ce91978c8af4f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.755796 13242 ts_tablet_manager.cc:1909] T 6d04a1c5ee7b4d529fa2f4da4945ff09 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:52.756033 12921 leader_election.cc:341] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.756088 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 766) took 3196 ms. Trace:
I20240506 19:56:52.756160 13242 rpcz_store.cc:276] 0506 19:56:49.559511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.559575 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:52.720339 (+3160764us) ts_tablet_manager.cc:1120] Deleting tablet 6d04a1c5ee7b4d529fa2f4da4945ff09
0506 19:56:52.720350 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.724173 (+ 3823us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.755253 (+ 31080us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.756070 (+ 817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3160570}
I20240506 19:56:52.756572 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 770) took 3158 ms. Trace:
I20240506 19:56:52.756644 13242 rpcz_store.cc:276] 0506 19:56:49.598561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.598627 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:52.756455 (+3157828us) ts_tablet_manager.cc:1120] Deleting tablet 979d789bc89a435b9e6cd8d6dc9c2037
0506 19:56:52.756467 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.756556 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3157629}
I20240506 19:56:52.756704 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 6d04a1c5ee7b4d529fa2f4da4945ff09 (table default.Table_bece2f866f8449ad8ef2f48007dc14ef [id=4021900d99ce4302b2de9b33a687e94f]) successfully deleted
I20240506 19:56:52.756975 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 772) took 3124 ms. Trace:
I20240506 19:56:52.757041 13242 rpcz_store.cc:276] 0506 19:56:49.632551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.655280 (+ 22729us) service_pool.cc:225] Handling call
0506 19:56:52.756878 (+3101598us) ts_tablet_manager.cc:1120] Deleting tablet 744864043f9347e0a13f8c2a7d6b147f
0506 19:56:52.756885 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.756962 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3101425}
I20240506 19:56:52.757328 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 788) took 2919 ms. Trace:
I20240506 19:56:52.757393 13242 rpcz_store.cc:276] 0506 19:56:49.837967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.853819 (+ 15852us) service_pool.cc:225] Handling call
0506 19:56:52.757215 (+2903396us) ts_tablet_manager.cc:1120] Deleting tablet f3bbf5a4326448d684fcb3f138dcf96d
0506 19:56:52.757222 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.757313 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2902836}
W20240506 19:56:52.757485 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 979d789bc89a435b9e6cd8d6dc9c2037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 979d789bc89a435b9e6cd8d6dc9c2037 already in progress: creating tablet
I20240506 19:56:52.757622 13242 tablet_replica.cc:330] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:52.757755 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 744864043f9347e0a13f8c2a7d6b147f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 744864043f9347e0a13f8c2a7d6b147f already in progress: creating tablet
I20240506 19:56:52.757822 13242 raft_consensus.cc:2229] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.757974 13242 raft_consensus.cc:2258] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:52.758095 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f3bbf5a4326448d684fcb3f138dcf96d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bbf5a4326448d684fcb3f138dcf96d already in progress: creating tablet
I20240506 19:56:52.758690 14314 leader_election.cc:288] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.759339 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f4861a01404663870ce91978c8af4f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.759521 13123 raft_consensus.cc:3037] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.762269 13242 ts_tablet_manager.cc:1875] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.762799 13123 raft_consensus.cc:2454] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.763258 12922 leader_election.cc:302] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.763751 14314 raft_consensus.cc:2781] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.763985 14314 raft_consensus.cc:686] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.764333 14314 consensus_queue.cc:237] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.765033 13245 ts_tablet_manager.cc:1367] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.000s
I20240506 19:56:52.765393 14080 raft_consensus.cc:482] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.765506 14080 raft_consensus.cc:504] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.766052 14080 leader_election.cc:288] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.766602 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c63d8995346d78633154a8f0d4c2b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.767103 12921 leader_election.cc:341] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.768098 13242 ts_tablet_manager.cc:1888] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.768144 13249 tablet_bootstrap.cc:492] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.768204 13242 log.cc:1196] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/db13b58d17c94e0f82052f823915d9af
I20240506 19:56:52.768270 13249 ts_tablet_manager.cc:1367] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240506 19:56:52.768497 13242 ts_tablet_manager.cc:1909] T db13b58d17c94e0f82052f823915d9af P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.768785 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 789) took 2913 ms. Trace:
I20240506 19:56:52.768857 13242 rpcz_store.cc:276] 0506 19:56:49.855607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.855680 (+ 73us) service_pool.cc:225] Handling call
0506 19:56:52.757584 (+2901904us) ts_tablet_manager.cc:1120] Deleting tablet db13b58d17c94e0f82052f823915d9af
0506 19:56:52.757592 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.765215 (+ 7623us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.768004 (+ 2789us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.768768 (+ 764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2901692}
I20240506 19:56:52.769060 13249 raft_consensus.cc:348] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.769091 13245 raft_consensus.cc:348] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.769228 13249 raft_consensus.cc:374] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.769285 13249 raft_consensus.cc:729] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.769280 13245 raft_consensus.cc:374] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.769387 13245 raft_consensus.cc:729] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.769506 13249 consensus_queue.cc:260] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.769650 13245 consensus_queue.cc:260] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.770006 13249 ts_tablet_manager.cc:1398] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:52.770257 13249 tablet_bootstrap.cc:492] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.770295 13245 ts_tablet_manager.cc:1398] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:56:52.772302 13249 tablet_bootstrap.cc:654] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.773258 14080 raft_consensus.cc:482] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.773382 14080 raft_consensus.cc:504] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.773933 14080 leader_election.cc:288] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:52.774360 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet db13b58d17c94e0f82052f823915d9af (table default.Table_bb8517f5ef13428c9dcfc8c72d0cab0e [id=21f83b4858344bf9959aee15e02a7bf7]) successfully deleted
I20240506 19:56:52.775046 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e90b93023e40b3bafbddf7bbcc10b2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.775924 12921 leader_election.cc:341] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:52.775965 13058 consensus_peers.cc:485] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:52.768255 12922 leader_election.cc:341] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 472c63d8995346d78633154a8f0d4c2b
I20240506 19:56:52.776594 12922 leader_election.cc:302] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.777070 14080 raft_consensus.cc:2726] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.778084 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e90b93023e40b3bafbddf7bbcc10b2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.778288 13120 raft_consensus.cc:2454] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.778721 12922 leader_election.cc:302] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.779101 14080 raft_consensus.cc:2781] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.779186 14080 raft_consensus.cc:482] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.779243 14080 raft_consensus.cc:3037] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.782035 14080 raft_consensus.cc:504] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.769239 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 795) took 2763 ms. Trace:
I20240506 19:56:52.783159 13242 rpcz_store.cc:276] 0506 19:56:50.006027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.006573 (+ 546us) service_pool.cc:225] Handling call
0506 19:56:52.769125 (+2762552us) ts_tablet_manager.cc:1120] Deleting tablet 8321457717fd4ad28ae1292d9e9a8669
0506 19:56:52.769133 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.769223 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2762328}
I20240506 19:56:52.783260 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e90b93023e40b3bafbddf7bbcc10b2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.783601 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 794) took 2777 ms. Trace:
W20240506 19:56:52.783731 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8321457717fd4ad28ae1292d9e9a8669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8321457717fd4ad28ae1292d9e9a8669 already in progress: creating tablet
I20240506 19:56:52.783691 13242 rpcz_store.cc:276] 0506 19:56:50.005883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.006829 (+ 946us) service_pool.cc:225] Handling call
0506 19:56:52.783475 (+2776646us) ts_tablet_manager.cc:1120] Deleting tablet 46fd9906504e4b95a5b05b01dc3ea1c0
0506 19:56:52.783485 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.783583 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2776517}
I20240506 19:56:52.783948 13242 tablet_replica.cc:330] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
W20240506 19:56:52.784018 12921 leader_election.cc:341] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.784144 13242 raft_consensus.cc:2229] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.784358 13242 raft_consensus.cc:2258] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:52.784461 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 46fd9906504e4b95a5b05b01dc3ea1c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46fd9906504e4b95a5b05b01dc3ea1c0 already in progress: creating tablet
I20240506 19:56:52.784591 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e90b93023e40b3bafbddf7bbcc10b2" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.784775 13120 raft_consensus.cc:3037] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.785024 13242 ts_tablet_manager.cc:1875] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.787649 13249 tablet_bootstrap.cc:492] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.787704 13120 raft_consensus.cc:2454] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.787808 13249 ts_tablet_manager.cc:1367] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240506 19:56:52.788216 12922 leader_election.cc:302] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.788481 14080 leader_election.cc:288] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:52.788542 13249 raft_consensus.cc:348] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.788764 14080 raft_consensus.cc:2781] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.788875 14080 raft_consensus.cc:686] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.788779 13249 raft_consensus.cc:374] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.789098 14080 consensus_queue.cc:237] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.789942 13249 raft_consensus.cc:729] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.794462 13249 consensus_queue.cc:260] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.795094 13249 ts_tablet_manager.cc:1398] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:52.795384 13249 tablet_bootstrap.cc:492] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.797494 13249 tablet_bootstrap.cc:654] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.799945 13244 tablet_bootstrap.cc:492] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:52.800164 13244 ts_tablet_manager.cc:1367] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.183s user 0.003s sys 0.002s
I20240506 19:56:52.801189 13244 raft_consensus.cc:348] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.801415 13244 raft_consensus.cc:374] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.801553 13244 raft_consensus.cc:729] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.801945 13244 consensus_queue.cc:260] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.803642 13244 ts_tablet_manager.cc:1398] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:52.809907 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b24176d9d941d4a9dfb97e24b91ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:52.816905 14383 raft_consensus.cc:482] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.817098 14383 raft_consensus.cc:504] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.818876 14383 leader_election.cc:288] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.819185 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47449a5a504f939e131743d94433ee" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:52.819408 13123 raft_consensus.cc:2454] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:52.819833 12922 leader_election.cc:302] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:52.820219 14383 raft_consensus.cc:2781] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.820302 14383 raft_consensus.cc:482] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:52.820374 14383 raft_consensus.cc:3037] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.823228 14383 raft_consensus.cc:504] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.824766 14383 leader_election.cc:288] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.825037 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47449a5a504f939e131743d94433ee" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:52.825241 13123 raft_consensus.cc:3037] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.827875 13123 raft_consensus.cc:2454] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:52.828318 12922 leader_election.cc:302] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:52.833907 13242 ts_tablet_manager.cc:1888] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240506 19:56:52.836937 14080 raft_consensus.cc:2781] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.837093 14080 raft_consensus.cc:686] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:52.837538 14080 consensus_queue.cc:237] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.839114 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 871) took 2133 ms. Trace:
I20240506 19:56:52.839236 13246 rpcz_store.cc:276] 0506 19:56:50.705737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.709348 (+ 3611us) service_pool.cc:225] Handling call
0506 19:56:52.715341 (+2005993us) ts_tablet_manager.cc:1120] Deleting tablet 796beb8e0a2c41d698bd4c05796263f1
0506 19:56:52.715355 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.720607 (+ 5252us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.735160 (+ 14553us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.839084 (+103924us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005688}
I20240506 19:56:52.839655 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 876) took 1998 ms. Trace:
I20240506 19:56:52.839736 13246 rpcz_store.cc:276] 0506 19:56:50.841269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.841936 (+ 667us) service_pool.cc:225] Handling call
0506 19:56:52.839528 (+1997592us) ts_tablet_manager.cc:1120] Deleting tablet 7f8989f02f25418ea9f824b3e5eeddd6
0506 19:56:52.839537 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.839639 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997278}
I20240506 19:56:52.839958 13246 tablet_replica.cc:330] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.840169 13246 raft_consensus.cc:2229] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.840312 13246 raft_consensus.cc:2258] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.840817 13246 ts_tablet_manager.cc:1875] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.844282 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_dbd73da3dbb94cc59750f8b4d27a46ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.844914 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbd73da3dbb94cc59750f8b4d27a46ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.845109 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:52.845571 13245 tablet_bootstrap.cc:492] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.846836 13245 tablet_bootstrap.cc:654] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.847360 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a94082868b214538890b435c8c101305" } new_table_name: "default.Table_1a6834c34ebb414dbedf9e1560ac7462" modify_external_catalogs: true
I20240506 19:56:52.849655 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5e49e35a08cb438ebd5204ba40131503" } modify_external_catalogs: true
I20240506 19:56:52.849838 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5e49e35a08cb438ebd5204ba40131503" } modify_external_catalogs: true
I20240506 19:56:52.850755 13246 ts_tablet_manager.cc:1888] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.850872 13246 log.cc:1196] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/9bb90bf767c94d73ace6411cfd31d65c
I20240506 19:56:52.851147 13246 ts_tablet_manager.cc:1909] T 9bb90bf767c94d73ace6411cfd31d65c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:52.853272 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8ce7f4ceb9d40009e9c75543b1fc545 consensus state has no leader
I20240506 19:56:52.853762 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b2476ed5d80041e68f12fa12ddef7008" } modify_external_catalogs: true
I20240506 19:56:52.854008 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b2476ed5d80041e68f12fa12ddef7008" } modify_external_catalogs: true
I20240506 19:56:52.854961 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1b37b87a1904426883b754783f6d5c2e" } modify_external_catalogs: true
I20240506 19:56:52.855130 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1b37b87a1904426883b754783f6d5c2e" } modify_external_catalogs: true
I20240506 19:56:52.855999 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b879654fd2304197a107fc833fd81e21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:52.856616 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b879654fd2304197a107fc833fd81e21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.858933 13242 log.cc:1196] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/41faccd54f3e49ddb48ea5234f6cdab5
I20240506 19:56:52.859390 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 877) took 2017 ms. Trace:
I20240506 19:56:52.859491 13246 rpcz_store.cc:276] 0506 19:56:50.841869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.841927 (+ 58us) service_pool.cc:225] Handling call
0506 19:56:52.839922 (+1997995us) ts_tablet_manager.cc:1120] Deleting tablet 9bb90bf767c94d73ace6411cfd31d65c
0506 19:56:52.839929 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.843892 (+ 3963us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.850644 (+ 6752us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.859368 (+ 8724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997544}
I20240506 19:56:52.860105 13242 ts_tablet_manager.cc:1909] T 41faccd54f3e49ddb48ea5234f6cdab5 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.861578 14383 raft_consensus.cc:482] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.861754 14383 raft_consensus.cc:504] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.862519 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 874) took 2022 ms. Trace:
I20240506 19:56:52.862522 14383 leader_election.cc:288] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:52.862614 13246 rpcz_store.cc:276] 0506 19:56:50.839632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.842415 (+ 2783us) service_pool.cc:225] Handling call
0506 19:56:52.862386 (+2019971us) ts_tablet_manager.cc:1120] Deleting tablet a03f574aec984b83ab596e92a6f323fc
0506 19:56:52.862398 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.862499 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019811}
W20240506 19:56:52.863492 12922 leader_election.cc:341] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:52.863751 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 879) took 2020 ms. Trace:
I20240506 19:56:52.864184 13246 rpcz_store.cc:276] 0506 19:56:50.843160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.843268 (+ 108us) service_pool.cc:225] Handling call
0506 19:56:52.863644 (+2020376us) ts_tablet_manager.cc:1120] Deleting tablet 996eb97639464e0bb976f2b1ecbbb73b
0506 19:56:52.863653 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.863737 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2020217}
I20240506 19:56:52.865583 13246 tablet_replica.cc:330] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.865800 13246 raft_consensus.cc:2229] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.865762 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971db6ac24d54efca95695ed1c90d4ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.865953 13246 raft_consensus.cc:2258] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.865931 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47449a5a504f939e131743d94433ee" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.866364 13244 tablet_bootstrap.cc:492] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:52.866439 13246 ts_tablet_manager.cc:1875] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:52.866580 12921 leader_election.cc:341] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.866683 12921 leader_election.cc:302] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:52.867074 12921 leader_election.cc:341] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.865754 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47449a5a504f939e131743d94433ee" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.867803 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 796) took 2829 ms. Trace:
W20240506 19:56:52.867851 12921 leader_election.cc:341] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.867897 13242 rpcz_store.cc:276] 0506 19:56:50.038347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.038425 (+ 78us) service_pool.cc:225] Handling call
0506 19:56:52.783907 (+2745482us) ts_tablet_manager.cc:1120] Deleting tablet 41faccd54f3e49ddb48ea5234f6cdab5
0506 19:56:52.783916 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.830620 (+ 46704us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.833807 (+ 3187us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.867785 (+ 33978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745088}
I20240506 19:56:52.868300 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 41faccd54f3e49ddb48ea5234f6cdab5 (table default.Table_6ef46f8805ee4fdfa0250b1af59ef09d [id=f13c2a81c2384d5bb7dbe0cef8718f1e]) successfully deleted
I20240506 19:56:52.868620 13242 tablet_replica.cc:330] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:52.868701 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:52.868810 13242 raft_consensus.cc:2229] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.868948 13242 raft_consensus.cc:2258] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.869480 13242 ts_tablet_manager.cc:1875] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.874068 13245 tablet_bootstrap.cc:492] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.874222 13245 ts_tablet_manager.cc:1367] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.004s
I20240506 19:56:52.874740 14211 raft_consensus.cc:482] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.874874 14211 raft_consensus.cc:504] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.875262 13245 raft_consensus.cc:348] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.875427 13245 raft_consensus.cc:374] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.875512 14211 leader_election.cc:288] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.875474 13245 raft_consensus.cc:729] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.876163 13245 consensus_queue.cc:260] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.876740 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234e4f3439e84b5fbb2f7bab1dbd0822" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.877221 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234e4f3439e84b5fbb2f7bab1dbd0822" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:52.877876 13058 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.878118 13245 ts_tablet_manager.cc:1398] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:52.878378 13245 tablet_bootstrap.cc:492] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.879619 13245 tablet_bootstrap.cc:654] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:52.880245 13057 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.880357 13057 leader_election.cc:302] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.880571 14211 raft_consensus.cc:2726] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.883543 14274 raft_consensus.cc:2726] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:52.860563 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 796beb8e0a2c41d698bd4c05796263f1 (table default.Table_58c604462fcb4b9cab62687d64877b02 [id=923944e6f77349c3b09903ba96a887fc]) successfully deleted
W20240506 19:56:52.886584 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7f8989f02f25418ea9f824b3e5eeddd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f8989f02f25418ea9f824b3e5eeddd6 already in progress: creating tablet
I20240506 19:56:52.889503 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 9bb90bf767c94d73ace6411cfd31d65c (table default.Table_a410467878c2481f86118d12ccd06f28 [id=f339c5ce6df04d839431b2475a6c1d8d]) successfully deleted
W20240506 19:56:52.892503 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a03f574aec984b83ab596e92a6f323fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a03f574aec984b83ab596e92a6f323fc already in progress: creating tablet
I20240506 19:56:52.895205 13249 tablet_bootstrap.cc:492] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:52.896266 13249 ts_tablet_manager.cc:1367] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.000s
W20240506 19:56:52.897779 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 996eb97639464e0bb976f2b1ecbbb73b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996eb97639464e0bb976f2b1ecbbb73b already in progress: creating tablet
I20240506 19:56:52.899063 13245 tablet_bootstrap.cc:492] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.899199 13245 ts_tablet_manager.cc:1367] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20240506 19:56:52.899727 13244 tablet_bootstrap.cc:654] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.897104 13249 raft_consensus.cc:348] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.902580 13249 raft_consensus.cc:374] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.902639 13245 raft_consensus.cc:348] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.902853 13245 raft_consensus.cc:374] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.902904 13245 raft_consensus.cc:729] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.902948 13249 raft_consensus.cc:729] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.903106 13245 consensus_queue.cc:260] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:52.903290 13249 consensus_queue.cc:260] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.906607 13245 ts_tablet_manager.cc:1398] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:52.906915 13245 tablet_bootstrap.cc:492] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.908255 13245 tablet_bootstrap.cc:654] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.909051 13249 ts_tablet_manager.cc:1398] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240506 19:56:52.909530 13249 tablet_bootstrap.cc:492] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:52.920264 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c047db21c5bb45a8a1a20ce3ade15da0" } new_table_name: "default.Table_8ab9abf5e45d49b1a3377c34813da069" modify_external_catalogs: true
I20240506 19:56:52.921348 13246 ts_tablet_manager.cc:1888] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.921483 13246 log.cc:1196] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b3e50bde3564493eb7267acb42cb8592
W20240506 19:56:52.921541 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbb1c433315c4bdbac4419ef8345c075 consensus state has no leader
I20240506 19:56:52.921802 13246 ts_tablet_manager.cc:1909] T b3e50bde3564493eb7267acb42cb8592 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.922116 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_31d441acbe354b93b7056bee4a33a55a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:52.922474 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 875) took 2081 ms. Trace:
I20240506 19:56:52.922552 13246 rpcz_store.cc:276] 0506 19:56:50.840721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.861333 (+ 20612us) service_pool.cc:225] Handling call
0506 19:56:52.865528 (+2004195us) ts_tablet_manager.cc:1120] Deleting tablet b3e50bde3564493eb7267acb42cb8592
0506 19:56:52.865539 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.892928 (+ 27389us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.910626 (+ 17698us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.922452 (+ 11826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003947}
W20240506 19:56:52.922698 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d441acbe354b93b7056bee4a33a55a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:52.923440 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 878) took 2080 ms. Trace:
I20240506 19:56:52.923516 13246 rpcz_store.cc:276] 0506 19:56:50.842508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.863467 (+ 20959us) service_pool.cc:225] Handling call
0506 19:56:52.923314 (+2059847us) ts_tablet_manager.cc:1120] Deleting tablet 5b15189e5cb54684a0f62308c2e5f131
0506 19:56:52.923324 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.923423 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059620}
I20240506 19:56:52.923918 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 905) took 1637 ms. Trace:
I20240506 19:56:52.923980 13246 rpcz_store.cc:276] 0506 19:56:51.286200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.289743 (+ 3543us) service_pool.cc:225] Handling call
0506 19:56:52.923827 (+1634084us) ts_tablet_manager.cc:1120] Deleting tablet 5bbcc40911034f319f489931015ca257
0506 19:56:52.923835 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.923906 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1633893}
I20240506 19:56:52.924476 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 2417a535329341d8bb9f8218770f890d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:52.924876 13246 tablet_replica.cc:330] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.925100 13246 raft_consensus.cc:2229] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.925256 13246 raft_consensus.cc:2258] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.925808 13246 ts_tablet_manager.cc:1875] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.926484 13249 tablet_bootstrap.cc:654] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.913347 14080 consensus_queue.cc:1035] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:52.929850 13245 tablet_bootstrap.cc:492] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.929983 13245 ts_tablet_manager.cc:1367] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20240506 19:56:52.930711 13245 raft_consensus.cc:348] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.930912 13245 raft_consensus.cc:374] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.930960 13245 raft_consensus.cc:729] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.931151 13245 consensus_queue.cc:260] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.931653 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:52.931767 13245 ts_tablet_manager.cc:1398] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:52.932034 13245 tablet_bootstrap.cc:492] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:52.936210 13245 tablet_bootstrap.cc:654] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.947260 14211 raft_consensus.cc:482] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.947448 14211 raft_consensus.cc:504] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.948092 14211 leader_election.cc:288] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.948710 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17da01620b8b41008421b977455e0c9a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:52.948710 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17da01620b8b41008421b977455e0c9a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:52.948916 12982 raft_consensus.cc:2454] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:52.949344 13057 leader_election.cc:302] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:52.949620 14211 raft_consensus.cc:2781] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:52.949718 14211 raft_consensus.cc:482] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240506 19:56:52.949759 13058 leader_election.cc:341] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.949776 14211 raft_consensus.cc:3037] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.951000 14331 raft_consensus.cc:482] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.951186 14331 raft_consensus.cc:504] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.951910 14331 leader_election.cc:288] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.952401 13246 ts_tablet_manager.cc:1888] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.952499 13246 log.cc:1196] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2b3268094cbf481698cb373c1bf6dfba
I20240506 19:56:52.952610 14211 raft_consensus.cc:504] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.952834 13246 ts_tablet_manager.cc:1909] T 2b3268094cbf481698cb373c1bf6dfba P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.953130 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 906) took 1666 ms. Trace:
W20240506 19:56:52.953930 12783 leader_election.cc:341] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: fdab4110236e4f5c8690f6d963ff27f7
I20240506 19:56:52.954123 14211 leader_election.cc:288] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:52.954679 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17da01620b8b41008421b977455e0c9a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:52.954792 13245 tablet_bootstrap.cc:492] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.954907 13245 ts_tablet_manager.cc:1367] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
W20240506 19:56:52.955300 13058 leader_election.cc:341] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:52.955549 13245 raft_consensus.cc:348] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.955734 13245 raft_consensus.cc:374] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.955793 13245 raft_consensus.cc:729] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.955996 13245 consensus_queue.cc:260] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.956466 13245 ts_tablet_manager.cc:1398] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:52.955659 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b3e50bde3564493eb7267acb42cb8592 (table default.Table_95be5bd2c51b4613b0152034aae6350d [id=b8f3e6ecf02f4dd299adfa1d7464e53f]) successfully deleted
I20240506 19:56:52.956844 13245 tablet_bootstrap.cc:492] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:52.957751 12781 leader_election.cc:341] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: fdab4110236e4f5c8690f6d963ff27f7
I20240506 19:56:52.957760 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17da01620b8b41008421b977455e0c9a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:52.957866 12781 leader_election.cc:302] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:52.957931 12982 raft_consensus.cc:3037] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:52.958281 13245 tablet_bootstrap.cc:654] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:52.958261 14331 raft_consensus.cc:2726] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:52.959059 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5b15189e5cb54684a0f62308c2e5f131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b15189e5cb54684a0f62308c2e5f131 already in progress: creating tablet
I20240506 19:56:52.953197 13246 rpcz_store.cc:276] 0506 19:56:51.286295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.293015 (+ 6720us) service_pool.cc:225] Handling call
0506 19:56:52.924834 (+1631819us) ts_tablet_manager.cc:1120] Deleting tablet 2b3268094cbf481698cb373c1bf6dfba
0506 19:56:52.924842 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.940774 (+ 15932us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.952290 (+ 11516us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.953114 (+ 824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631587}
W20240506 19:56:52.959329 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5bbcc40911034f319f489931015ca257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bbcc40911034f319f489931015ca257 already in progress: creating tablet
I20240506 19:56:52.960479 12982 raft_consensus.cc:2454] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:52.960602 13242 ts_tablet_manager.cc:1888] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.960721 13242 log.cc:1196] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/517c09e3bf7046ccba6ed25b08e3460e
I20240506 19:56:52.961023 13242 ts_tablet_manager.cc:1909] T 517c09e3bf7046ccba6ed25b08e3460e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:52.960997 13057 leader_election.cc:302] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:52.961254 14211 raft_consensus.cc:2781] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:52.961529 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 904) took 1675 ms. Trace:
I20240506 19:56:52.961617 13246 rpcz_store.cc:276] 0506 19:56:51.286083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.289655 (+ 3572us) service_pool.cc:225] Handling call
0506 19:56:52.961410 (+1671755us) ts_tablet_manager.cc:1120] Deleting tablet 9c93117bdcf44c719c2e3588de538e96
0506 19:56:52.961420 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.961511 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665449}
I20240506 19:56:52.962088 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 903) took 1676 ms. Trace:
I20240506 19:56:52.962167 13246 rpcz_store.cc:276] 0506 19:56:51.285988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.289574 (+ 3586us) service_pool.cc:225] Handling call
0506 19:56:52.961991 (+1672417us) ts_tablet_manager.cc:1120] Deleting tablet 2a3291123a1e42af80dbfb578c855618
0506 19:56:52.962000 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.962073 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1661484}
I20240506 19:56:52.962482 13246 tablet_replica.cc:330] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.962669 13246 raft_consensus.cc:2229] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.962862 13246 raft_consensus.cc:2258] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.963338 13246 ts_tablet_manager.cc:1875] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.970424 13245 tablet_bootstrap.cc:492] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:52.970582 13245 ts_tablet_manager.cc:1367] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240506 19:56:52.971334 13245 raft_consensus.cc:348] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.971562 13245 raft_consensus.cc:374] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.971621 13245 raft_consensus.cc:729] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.971662 13244 tablet_bootstrap.cc:492] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:52.971913 13244 ts_tablet_manager.cc:1367] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.002s
I20240506 19:56:52.971884 13245 consensus_queue.cc:260] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:52.980185 12921 consensus_peers.cc:485] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:52.982133 13244 raft_consensus.cc:348] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.982380 13244 raft_consensus.cc:374] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:52.982455 13244 raft_consensus.cc:729] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:52.982709 13244 consensus_queue.cc:260] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.990893 13245 ts_tablet_manager.cc:1398] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.020s user 0.003s sys 0.001s
I20240506 19:56:52.991956 13246 ts_tablet_manager.cc:1888] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:52.992098 13246 log.cc:1196] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2c96f014293242be9640515697c0733b
I20240506 19:56:52.992424 13246 ts_tablet_manager.cc:1909] T 2c96f014293242be9640515697c0733b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:52.992717 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 914) took 1558 ms. Trace:
I20240506 19:56:52.992733 14211 raft_consensus.cc:686] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:52.992789 13246 rpcz_store.cc:276] 0506 19:56:51.434103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.434165 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:52.962443 (+1528278us) ts_tablet_manager.cc:1120] Deleting tablet 2c96f014293242be9640515697c0733b
0506 19:56:52.962451 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.973191 (+ 10740us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.991837 (+ 18646us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.992700 (+ 863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528051}
I20240506 19:56:52.993181 14211 consensus_queue.cc:237] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.995605 13244 ts_tablet_manager.cc:1398] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240506 19:56:52.997244 13246 tablet_replica.cc:330] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:52.997539 13246 raft_consensus.cc:2229] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:52.997726 13246 raft_consensus.cc:2258] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:52.998215 14080 raft_consensus.cc:482] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:52.998427 14080 raft_consensus.cc:504] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:52.998570 13246 ts_tablet_manager.cc:1875] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:52.998951 13244 tablet_bootstrap.cc:492] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:52.999658 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b59ad067c274fddbba5f71e89480dec" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.000321 12921 leader_election.cc:341] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.000648 13244 tablet_bootstrap.cc:654] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.001327 14080 leader_election.cc:288] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:53.001816 12922 leader_election.cc:341] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:53.002058 12922 leader_election.cc:302] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.002513 14080 raft_consensus.cc:2726] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.004247 14401 raft_consensus.cc:482] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.005697 14080 raft_consensus.cc:482] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.005895 14080 raft_consensus.cc:504] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.005074 14401 raft_consensus.cc:504] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.007010 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeae70e0d1144ea9e3e0f8727c15e6b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.007273 13124 raft_consensus.cc:2454] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:53.007457 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeae70e0d1144ea9e3e0f8727c15e6b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.008090 12921 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.008459 12922 leader_election.cc:302] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.008754 14314 raft_consensus.cc:2781] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.008855 14314 raft_consensus.cc:482] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.008961 14314 raft_consensus.cc:3037] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.011528 14314 raft_consensus.cc:504] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.012595 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeae70e0d1144ea9e3e0f8727c15e6b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:53.012826 13124 raft_consensus.cc:3037] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.013092 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeae70e0d1144ea9e3e0f8727c15e6b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:53.013711 12921 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.013898 14314 leader_election.cc:288] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.014083 14080 leader_election.cc:288] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.016690 13124 raft_consensus.cc:2454] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:53.016899 13249 tablet_bootstrap.cc:492] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.017732 14401 leader_election.cc:288] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.017815 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.018347 14401 raft_consensus.cc:482] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:53.018431 13058 leader_election.cc:341] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.017205 12922 leader_election.cc:302] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.020603 14383 raft_consensus.cc:2781] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:53.020782 14383 raft_consensus.cc:686] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:53.021122 14383 consensus_queue.cc:237] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.026670 14211 raft_consensus.cc:482] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.027143 14211 raft_consensus.cc:504] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:56:53.031031 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88d16a40c7c24972861d582fb68d4da4 has no consensus state
W20240506 19:56:53.031798 12508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.253:36819, user_credentials={real_user=slave}} blocked reactor thread for 72056us
I20240506 19:56:53.033619 14331 raft_consensus.cc:482] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.033831 14331 raft_consensus.cc:504] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.034565 14331 leader_election.cc:288] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.034798 13245 tablet_bootstrap.cc:492] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:53.036016 12781 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:53.036304 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea1edc3350f3474c91f0828a99c2228f" } new_table_name: "default.Table_bc52ab9a06b149e999bf34c308e24839" modify_external_catalogs: true
I20240506 19:56:53.036887 13245 tablet_bootstrap.cc:654] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.039981 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 797) took 2968 ms. Trace:
I20240506 19:56:53.045490 14383 raft_consensus.cc:482] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.045817 14383 raft_consensus.cc:504] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.047995 13246 ts_tablet_manager.cc:1888] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.048074 13249 ts_tablet_manager.cc:1367] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.139s user 0.005s sys 0.005s
I20240506 19:56:53.048135 13246 log.cc:1196] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a9e90b93023e40b3bafbddf7bbcc10b2
I20240506 19:56:53.048475 13246 ts_tablet_manager.cc:1909] T a9e90b93023e40b3bafbddf7bbcc10b2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.048817 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 913) took 1614 ms. Trace:
I20240506 19:56:53.048764 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.048702 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_acd9eb17729b45d69cd7e0b55fb9e63a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:53.048902 13246 rpcz_store.cc:276] 0506 19:56:51.434001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.434261 (+ 260us) service_pool.cc:225] Handling call
0506 19:56:52.997167 (+1562906us) ts_tablet_manager.cc:1120] Deleting tablet a9e90b93023e40b3bafbddf7bbcc10b2
0506 19:56:52.997184 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.008684 (+ 11500us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.047873 (+ 39189us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.048797 (+ 924us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115,"tablet-delete.queue_time_us":1562749}
W20240506 19:56:53.049376 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acd9eb17729b45d69cd7e0b55fb9e63a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:53.049453 13057 leader_election.cc:341] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.049571 13057 leader_election.cc:302] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.049830 14398 raft_consensus.cc:2726] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.050544 13249 raft_consensus.cc:348] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.050715 13249 raft_consensus.cc:374] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.050774 13249 raft_consensus.cc:729] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.050956 13249 consensus_queue.cc:260] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.052076 13249 ts_tablet_manager.cc:1398] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240506 19:56:53.052368 13249 tablet_bootstrap.cc:492] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.052592 13246 tablet_replica.cc:330] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.052845 13246 raft_consensus.cc:2229] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.053018 13246 raft_consensus.cc:2258] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.047837 14383 leader_election.cc:288] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:53.054679 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c1b76d968cf047ab833ec4ec4af83697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.055411 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1b76d968cf047ab833ec4ec4af83697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.058087 13246 ts_tablet_manager.cc:1875] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.027024 14401 raft_consensus.cc:504] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.060178 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1160ab972124630a9be7abeefb4aa56" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.060385 14401 leader_election.cc:288] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:53.060732 13057 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.061884 14211 leader_election.cc:288] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.047010 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1615f6167aae4dc096b9f563ab5e6215" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.047407 14331 raft_consensus.cc:482] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.062460 14331 raft_consensus.cc:504] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.063128 14331 leader_election.cc:288] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.063582 13245 tablet_bootstrap.cc:492] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.063721 13245 ts_tablet_manager.cc:1367] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.003s
I20240506 19:56:53.064502 13245 raft_consensus.cc:348] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.064668 13245 raft_consensus.cc:374] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.064714 13245 raft_consensus.cc:729] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.064886 13245 consensus_queue.cc:260] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.065407 13245 ts_tablet_manager.cc:1398] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240506 19:56:53.069633 12921 leader_election.cc:341] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.069618 12781 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:53.060165 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1160ab972124630a9be7abeefb4aa56" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.061549 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6352d2c630204df3a24e299e167f92d4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.070219 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6352d2c630204df3a24e299e167f92d4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:53.070531 13058 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.070647 13058 leader_election.cc:302] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.070880 13245 tablet_bootstrap.cc:492] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:53.071049 13058 leader_election.cc:341] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.071342 14398 raft_consensus.cc:482] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.071446 14398 raft_consensus.cc:504] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.072034 14398 leader_election.cc:288] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.072172 14398 raft_consensus.cc:2726] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.072743 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3435c11854649b7daf74f3d8c33d7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.072988 13245 tablet_bootstrap.cc:654] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:53.073264 13058 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.073707 13057 leader_election.cc:341] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.073732 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3435c11854649b7daf74f3d8c33d7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.073853 13057 leader_election.cc:302] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:53.074414 13057 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.074555 13057 leader_election.cc:302] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.074908 14398 raft_consensus.cc:2726] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.075085 14398 raft_consensus.cc:2726] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:53.080595 12783 leader_election.cc:341] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:53.080745 12783 leader_election.cc:302] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:53.081369 14331 raft_consensus.cc:2726] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:53.082010 12783 leader_election.cc:341] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:53.082110 12783 leader_election.cc:302] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:53.082561 14331 raft_consensus.cc:2726] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.083134 14274 raft_consensus.cc:482] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.083431 14274 raft_consensus.cc:504] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.084594 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3268094cbf481698cb373c1bf6dfba" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.085107 12921 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.085381 12922 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2b3268094cbf481698cb373c1bf6dfba
I20240506 19:56:53.085479 12922 leader_election.cc:302] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.085685 14383 raft_consensus.cc:2726] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:53.086131 12922 leader_election.cc:341] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1615f6167aae4dc096b9f563ab5e6215
I20240506 19:56:53.086257 12922 leader_election.cc:302] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.086449 14383 raft_consensus.cc:2726] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.086638 13245 tablet_bootstrap.cc:492] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.086604 13242 rpcz_store.cc:276] 0506 19:56:50.071327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.071399 (+ 72us) service_pool.cc:225] Handling call
0506 19:56:52.868575 (+2797176us) ts_tablet_manager.cc:1120] Deleting tablet 517c09e3bf7046ccba6ed25b08e3460e
0506 19:56:52.868585 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:52.894260 (+ 25675us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:52.960480 (+ 66220us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.039956 (+ 79476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2791881}
I20240506 19:56:53.086809 13246 ts_tablet_manager.cc:1888] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.086848 13245 ts_tablet_manager.cc:1367] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240506 19:56:53.087103 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 517c09e3bf7046ccba6ed25b08e3460e (table default.Table_eeef43c95a8c42ad86d2f5f85d9ba6f6 [id=4b18d70e1c8847de81b7742f55f56260]) successfully deleted
I20240506 19:56:53.087466 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 798) took 2975 ms. Trace:
I20240506 19:56:53.087561 13242 rpcz_store.cc:276] 0506 19:56:50.111830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.112250 (+ 420us) service_pool.cc:225] Handling call
0506 19:56:53.086927 (+2974677us) ts_tablet_manager.cc:1120] Deleting tablet f078559df48f4d23869ddcb4b3fca9fc
0506 19:56:53.087337 (+ 410us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.087446 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2974458}
W20240506 19:56:53.087958 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f078559df48f4d23869ddcb4b3fca9fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f078559df48f4d23869ddcb4b3fca9fc already in progress: creating tablet
I20240506 19:56:53.088176 13242 tablet_replica.cc:330] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:53.088375 13242 raft_consensus.cc:2229] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.088779 13242 raft_consensus.cc:2258] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.088663 13245 raft_consensus.cc:348] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.088992 13245 raft_consensus.cc:374] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.089125 13245 raft_consensus.cc:729] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.089380 13242 ts_tablet_manager.cc:1875] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.086911 13246 log.cc:1196] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/8eb9296bb1104b4cbe279eecd3683796
I20240506 19:56:53.089818 13245 consensus_queue.cc:260] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.090669 13246 ts_tablet_manager.cc:1909] T 8eb9296bb1104b4cbe279eecd3683796 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.091324 13245 ts_tablet_manager.cc:1398] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:56:53.091454 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 912) took 1657 ms. Trace:
I20240506 19:56:53.091526 13246 rpcz_store.cc:276] 0506 19:56:51.433895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.434156 (+ 261us) service_pool.cc:225] Handling call
0506 19:56:53.052521 (+1618365us) ts_tablet_manager.cc:1120] Deleting tablet 8eb9296bb1104b4cbe279eecd3683796
0506 19:56:53.052537 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.061720 (+ 9183us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.086705 (+ 24985us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.091437 (+ 4732us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33,"tablet-delete.queue_time_us":1617974}
I20240506 19:56:53.093467 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 916) took 1488 ms. Trace:
I20240506 19:56:53.093536 13245 tablet_bootstrap.cc:492] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.093571 13246 rpcz_store.cc:276] 0506 19:56:51.604982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.605400 (+ 418us) service_pool.cc:225] Handling call
0506 19:56:53.091808 (+1486408us) ts_tablet_manager.cc:1120] Deleting tablet d43da1d12ce647dcbcd31263932b2f54
0506 19:56:53.093340 (+ 1532us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.093448 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1486194}
I20240506 19:56:53.094038 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 917) took 1482 ms. Trace:
I20240506 19:56:53.094122 13246 rpcz_store.cc:276] 0506 19:56:51.611569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.611618 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:53.093923 (+1482305us) ts_tablet_manager.cc:1120] Deleting tablet 691e312510a440898a58bec43ab26e8f
0506 19:56:53.093932 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.094024 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482124}
I20240506 19:56:53.094413 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 918) took 1436 ms. Trace:
I20240506 19:56:53.094494 13246 rpcz_store.cc:276] 0506 19:56:51.658036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.658116 (+ 80us) service_pool.cc:225] Handling call
0506 19:56:53.094327 (+1436211us) ts_tablet_manager.cc:1120] Deleting tablet af2741336d5a4be1b5306319fc20cdc6
0506 19:56:53.094336 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.094401 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1435856}
I20240506 19:56:53.094789 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 919) took 1435 ms. Trace:
I20240506 19:56:53.094870 13246 rpcz_store.cc:276] 0506 19:56:51.659476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.659543 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:53.094693 (+1435150us) ts_tablet_manager.cc:1120] Deleting tablet e6d37bb1c2144c4fa71aab6113b9f06e
0506 19:56:53.094700 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.094774 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434936}
I20240506 19:56:53.095176 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 920) took 1409 ms. Trace:
I20240506 19:56:53.095252 13246 rpcz_store.cc:276] 0506 19:56:51.685658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.685725 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:53.095085 (+1409360us) ts_tablet_manager.cc:1120] Deleting tablet c1a5a8dc9a2a4856b8b8f2aaac187c36
0506 19:56:53.095093 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.095164 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405245}
I20240506 19:56:53.095566 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 921) took 1406 ms. Trace:
I20240506 19:56:53.095638 13246 rpcz_store.cc:276] 0506 19:56:51.688820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.688880 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:53.095477 (+1406597us) ts_tablet_manager.cc:1120] Deleting tablet c8eeaa71ac61424796c6ffdd993de9db
0506 19:56:53.095486 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.095553 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405561}
I20240506 19:56:53.095957 13245 tablet_bootstrap.cc:654] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.095927 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 923) took 1339 ms. Trace:
I20240506 19:56:53.096767 13246 rpcz_store.cc:276] 0506 19:56:51.756137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.756199 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:53.095833 (+1339634us) ts_tablet_manager.cc:1120] Deleting tablet 15ed0c7b4569490b91c54d5bdb187e49
0506 19:56:53.095842 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.095912 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339405}
I20240506 19:56:53.097093 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 925) took 1328 ms. Trace:
I20240506 19:56:53.097162 13246 rpcz_store.cc:276] 0506 19:56:51.769130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.781481 (+ 12351us) service_pool.cc:225] Handling call
0506 19:56:53.096999 (+1315518us) ts_tablet_manager.cc:1120] Deleting tablet 6a7715dad8184493b085d891428ad44c
0506 19:56:53.097008 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.097080 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315282}
I20240506 19:56:53.097735 14401 consensus_queue.cc:1035] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:53.098933 12805 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 153) took 1064 ms. Trace:
I20240506 19:56:53.099049 12805 rpcz_store.cc:276] 0506 19:56:52.034523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.060708 (+ 26185us) service_pool.cc:225] Handling call
0506 19:56:53.098895 (+1038187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:53.102042 14383 raft_consensus.cc:482] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.102267 14383 raft_consensus.cc:504] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.102494 13246 tablet_replica.cc:330] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.102712 13246 raft_consensus.cc:2229] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.102871 13246 raft_consensus.cc:2258] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.103457 14383 leader_election.cc:288] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:53.103487 13246 ts_tablet_manager.cc:1875] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:53.106323 12922 leader_election.cc:341] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 8eb9296bb1104b4cbe279eecd3683796
I20240506 19:56:53.106554 14274 leader_election.cc:288] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:53.107225 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9c93117bdcf44c719c2e3588de538e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c93117bdcf44c719c2e3588de538e96 already in progress: creating tablet
W20240506 19:56:53.107597 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2a3291123a1e42af80dbfb578c855618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a3291123a1e42af80dbfb578c855618 already in progress: creating tablet
I20240506 19:56:53.107686 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9296bb1104b4cbe279eecd3683796" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.107795 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2c96f014293242be9640515697c0733b (table default.Table_b41fdc2ddabb43b6abd4d24fecd7447a [id=f67e8f35a154427f95ae0f771099e941]) successfully deleted
I20240506 19:56:53.108006 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a9e90b93023e40b3bafbddf7bbcc10b2 (table default.Table_6b562516d9b845789bf9d901c015e37c [id=11d0ad6e93ef421182aaac061edb8501]) successfully deleted
I20240506 19:56:53.108181 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 8eb9296bb1104b4cbe279eecd3683796 (table default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]) successfully deleted
W20240506 19:56:53.108255 12921 leader_election.cc:341] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.108351 12921 leader_election.cc:302] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:53.108399 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d43da1d12ce647dcbcd31263932b2f54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43da1d12ce647dcbcd31263932b2f54 already in progress: creating tablet
W20240506 19:56:53.108606 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 691e312510a440898a58bec43ab26e8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691e312510a440898a58bec43ab26e8f already in progress: creating tablet
W20240506 19:56:53.108803 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet af2741336d5a4be1b5306319fc20cdc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af2741336d5a4be1b5306319fc20cdc6 already in progress: creating tablet
I20240506 19:56:53.109164 14274 raft_consensus.cc:2726] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:53.109428 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet e6d37bb1c2144c4fa71aab6113b9f06e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6d37bb1c2144c4fa71aab6113b9f06e already in progress: creating tablet
W20240506 19:56:53.109725 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 already in progress: creating tablet
W20240506 19:56:53.109943 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c8eeaa71ac61424796c6ffdd993de9db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8eeaa71ac61424796c6ffdd993de9db already in progress: creating tablet
W20240506 19:56:53.110160 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 15ed0c7b4569490b91c54d5bdb187e49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ed0c7b4569490b91c54d5bdb187e49 already in progress: creating tablet
W20240506 19:56:53.110406 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 6a7715dad8184493b085d891428ad44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a7715dad8184493b085d891428ad44c already in progress: creating tablet
I20240506 19:56:53.116883 13249 tablet_bootstrap.cc:654] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.119194 13246 ts_tablet_manager.cc:1888] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.119345 13246 log.cc:1196] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:53.119689 13246 ts_tablet_manager.cc:1909] T 9df3ab9dff5148f9bb335a7209b29bbc P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.120666 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 927) took 1312 ms. Trace:
I20240506 19:56:53.120790 13246 rpcz_store.cc:276] 0506 19:56:51.808575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.815263 (+ 6688us) service_pool.cc:225] Handling call
0506 19:56:53.097371 (+1282108us) ts_tablet_manager.cc:1120] Deleting tablet 9df3ab9dff5148f9bb335a7209b29bbc
0506 19:56:53.097897 (+ 526us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.115080 (+ 17183us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.119069 (+ 3989us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.120637 (+ 1568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280638}
I20240506 19:56:53.122910 13244 tablet_bootstrap.cc:492] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:53.123149 13244 ts_tablet_manager.cc:1367] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20240506 19:56:53.124028 13244 raft_consensus.cc:348] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.124337 13244 raft_consensus.cc:374] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.124449 13244 raft_consensus.cc:729] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.124799 13246 tablet_replica.cc:330] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.124730 13244 consensus_queue.cc:260] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.125000 13246 raft_consensus.cc:2229] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.125133 13246 raft_consensus.cc:2258] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.125245 14401 raft_consensus.cc:482] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.125417 14401 raft_consensus.cc:504] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.125674 13246 ts_tablet_manager.cc:1875] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.126878 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9443cdbaa3c40299b66d216673c4b1a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.127542 13244 ts_tablet_manager.cc:1398] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:53.127866 13244 tablet_bootstrap.cc:492] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:53.129608 13244 tablet_bootstrap.cc:654] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.137768 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 9df3ab9dff5148f9bb335a7209b29bbc (table default.Table_3d6b94b6306c4200bfde515a4b2c418c [id=6b539c4e5b2444daa0f9ae05ed98ff54]) successfully deleted
I20240506 19:56:53.138414 13246 ts_tablet_manager.cc:1888] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.138545 13246 log.cc:1196] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:56:53.138818 13246 ts_tablet_manager.cc:1909] T ceeae70e0d1144ea9e3e0f8727c15e6b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.139567 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 928) took 1307 ms. Trace:
I20240506 19:56:53.139665 13246 rpcz_store.cc:276] 0506 19:56:51.832319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.832404 (+ 85us) service_pool.cc:225] Handling call
0506 19:56:53.124740 (+1292336us) ts_tablet_manager.cc:1120] Deleting tablet ceeae70e0d1144ea9e3e0f8727c15e6b
0506 19:56:53.124752 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.128595 (+ 3843us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.138292 (+ 9697us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.139544 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291647}
W20240506 19:56:53.140698 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cae859ab30064c969dbf040da2c14524 has no consensus state
I20240506 19:56:53.140899 13244 tablet_bootstrap.cc:492] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:53.141024 13244 ts_tablet_manager.cc:1367] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240506 19:56:53.141750 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 932) took 1254 ms. Trace:
I20240506 19:56:53.141837 13246 rpcz_store.cc:276] 0506 19:56:51.887526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.887590 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:53.141414 (+1253824us) ts_tablet_manager.cc:1120] Deleting tablet 44ce8749846847108edf5f554139cbd9
0506 19:56:53.141425 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.141540 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253623}
I20240506 19:56:53.141793 13244 raft_consensus.cc:348] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.142005 13244 raft_consensus.cc:374] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.142071 13244 raft_consensus.cc:729] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.142254 13246 tablet_replica.cc:330] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.142302 13244 consensus_queue.cc:260] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.142436 13246 raft_consensus.cc:2229] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.142730 13246 raft_consensus.cc:2258] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.142796 13244 ts_tablet_manager.cc:1398] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:53.143270 13246 ts_tablet_manager.cc:1875] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.143278 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet ceeae70e0d1144ea9e3e0f8727c15e6b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:53.143788 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_78690cea0de8499bb46d757158d49954" } modify_external_catalogs: true
I20240506 19:56:53.143990 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_78690cea0de8499bb46d757158d49954" } modify_external_catalogs: true
I20240506 19:56:53.144809 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7297af847f20409aae62497c2b197328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.145363 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7297af847f20409aae62497c2b197328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:53.145493 13058 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.146800 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_69b33a179131461abb0c8a0c13a419dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.147274 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b33a179131461abb0c8a0c13a419dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.149094 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a5a5e193f245de96cc82986e292f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.150153 13245 tablet_bootstrap.cc:492] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.150436 13245 ts_tablet_manager.cc:1367] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.002s
I20240506 19:56:53.150000 14401 leader_election.cc:288] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.151322 13245 raft_consensus.cc:348] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.151628 13245 raft_consensus.cc:374] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.151562 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9443cdbaa3c40299b66d216673c4b1a" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.151698 13245 raft_consensus.cc:729] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.151909 13245 consensus_queue.cc:260] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.152510 13245 ts_tablet_manager.cc:1398] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20240506 19:56:53.152853 13057 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.153035 13057 leader_election.cc:302] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.153792 14401 raft_consensus.cc:2726] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.155766 13245 tablet_bootstrap.cc:492] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.157931 13245 tablet_bootstrap.cc:654] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.158406 13246 ts_tablet_manager.cc:1888] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.159190 13246 log.cc:1196] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:56:53.159528 13246 ts_tablet_manager.cc:1909] T 2a7c61244620487bbbac26dcdbf8a38c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.161760 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ceeae70e0d1144ea9e3e0f8727c15e6b (table default.Table_54f3105144e741119c67aec80c408fa5 [id=6369da651bef4fa58ae1c1711da138b8]) successfully deleted
W20240506 19:56:53.162089 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 44ce8749846847108edf5f554139cbd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ce8749846847108edf5f554139cbd9 already in progress: creating tablet
I20240506 19:56:53.162552 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 935) took 1135 ms. Trace:
I20240506 19:56:53.162647 13246 rpcz_store.cc:276] 0506 19:56:52.027180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.027228 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:53.142207 (+1114979us) ts_tablet_manager.cc:1120] Deleting tablet 2a7c61244620487bbbac26dcdbf8a38c
0506 19:56:53.142217 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.151533 (+ 9316us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.158303 (+ 6770us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.162527 (+ 4224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112803}
I20240506 19:56:53.163223 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 936) took 1135 ms. Trace:
I20240506 19:56:53.163304 13246 rpcz_store.cc:276] 0506 19:56:52.027516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.027566 (+ 50us) service_pool.cc:225] Handling call
0506 19:56:53.163090 (+1135524us) ts_tablet_manager.cc:1120] Deleting tablet 8a9b92795a7742caa2ee99ab7ca138bf
0506 19:56:53.163101 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.163206 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133631}
I20240506 19:56:53.163724 13246 tablet_replica.cc:330] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.163928 13246 raft_consensus.cc:2229] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.164109 13246 raft_consensus.cc:2258] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.164651 13246 ts_tablet_manager.cc:1875] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:53.165333 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 0a6db7d0ad024f53ad76920bcbc0248e (table default.Table_28ea2a7143b84c1c9eeb79da8bf201ad [id=44207be9fc564b5bba7a4513e0b3c1bd], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a6db7d0ad024f53ad76920bcbc0248e consensus state has no leader
I20240506 19:56:53.170013 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2a7c61244620487bbbac26dcdbf8a38c (table default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]) successfully deleted
I20240506 19:56:53.170883 13242 ts_tablet_manager.cc:1888] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.170991 13242 log.cc:1196] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d6e54543d1544b1c8cff3e890e787337
I20240506 19:56:53.171286 13242 ts_tablet_manager.cc:1909] T d6e54543d1544b1c8cff3e890e787337 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:53.171579 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8a9b92795a7742caa2ee99ab7ca138bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a9b92795a7742caa2ee99ab7ca138bf already in progress: creating tablet
I20240506 19:56:53.177709 13244 tablet_bootstrap.cc:492] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:53.178164 13249 tablet_bootstrap.cc:492] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.178295 13249 ts_tablet_manager.cc:1367] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.001s
I20240506 19:56:53.179041 13249 raft_consensus.cc:348] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.179232 13249 raft_consensus.cc:374] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.179279 13249 raft_consensus.cc:729] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.179564 13244 tablet_bootstrap.cc:654] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.179515 13249 consensus_queue.cc:260] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.181993 13249 ts_tablet_manager.cc:1398] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:56:53.182272 13249 tablet_bootstrap.cc:492] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.181273 13245 tablet_bootstrap.cc:492] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.183751 13245 ts_tablet_manager.cc:1367] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20240506 19:56:53.187150 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_68c8729d5e134c10be8d88e246c4e00d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.187767 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c8729d5e134c10be8d88e246c4e00d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.188407 13249 tablet_bootstrap.cc:654] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.189324 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1bb7f42c11b54b52b98a8181cdb076d3" } new_table_name: "default.Table_6013611882be454da8ec1ea823261f6a" modify_external_catalogs: true
I20240506 19:56:53.198217 14402 raft_consensus.cc:482] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.198668 14402 raft_consensus.cc:504] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.201790 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ce7f4ceb9d40009e9c75543b1fc545" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.202562 12982 raft_consensus.cc:2454] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:53.203150 13057 leader_election.cc:302] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:53.203449 14211 raft_consensus.cc:2781] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.203516 14211 raft_consensus.cc:482] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.203573 14211 raft_consensus.cc:3037] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.203487 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ce7f4ceb9d40009e9c75543b1fc545" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.204370 13058 leader_election.cc:341] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.205086 13244 tablet_bootstrap.cc:492] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:53.205231 13244 ts_tablet_manager.cc:1367] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240506 19:56:53.205406 14402 leader_election.cc:288] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.206167 14211 raft_consensus.cc:504] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.206667 13244 raft_consensus.cc:348] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.206852 13244 raft_consensus.cc:374] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.206912 13244 raft_consensus.cc:729] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.207130 13244 consensus_queue.cc:260] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.207682 13244 ts_tablet_manager.cc:1398] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:53.208436 13120 raft_consensus.cc:1261] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 2 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 19:56:53.209009 14383 consensus_queue.cc:1035] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:53.209125 14402 raft_consensus.cc:482] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.209327 14402 raft_consensus.cc:504] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.210201 13249 tablet_bootstrap.cc:492] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.210346 13249 ts_tablet_manager.cc:1367] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240506 19:56:53.211099 13249 raft_consensus.cc:348] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.211218 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7be9788df8e47d285bd932d078c6a8b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.211381 13249 raft_consensus.cc:374] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.211479 13249 raft_consensus.cc:729] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.211571 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 799) took 3095 ms. Trace:
I20240506 19:56:53.211671 13242 rpcz_store.cc:276] 0506 19:56:50.116511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.116571 (+ 60us) service_pool.cc:225] Handling call
0506 19:56:53.087798 (+2971227us) ts_tablet_manager.cc:1120] Deleting tablet d6e54543d1544b1c8cff3e890e787337
0506 19:56:53.087806 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.133663 (+ 45857us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.170760 (+ 37097us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.211551 (+ 40791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2970831}
I20240506 19:56:53.211771 13249 consensus_queue.cc:260] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.212146 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d6e54543d1544b1c8cff3e890e787337 (table default.Table_77ba2f5ca8614d1ebb987a154041901c [id=3045c42f00ac4ff19e0354945092c6d1]) successfully deleted
W20240506 19:56:53.213220 13058 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.213723 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ce7f4ceb9d40009e9c75543b1fc545" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:53.214202 13058 leader_election.cc:341] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.214380 13249 ts_tablet_manager.cc:1398] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:53.214661 13249 tablet_bootstrap.cc:492] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.216121 13249 tablet_bootstrap.cc:654] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.216121 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7be9788df8e47d285bd932d078c6a8b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:53.216683 13057 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.216794 13057 leader_election.cc:302] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.217025 14404 raft_consensus.cc:2726] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.220106 14211 leader_election.cc:288] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.221779 13246 ts_tablet_manager.cc:1888] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:53.221910 13246 log.cc:1196] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/67e8b8be953a4a5784f2d098fd6364e3
I20240506 19:56:53.222283 13246 ts_tablet_manager.cc:1909] T 67e8b8be953a4a5784f2d098fd6364e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.222569 13245 raft_consensus.cc:348] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.222806 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 260d7ab458794df496012cb80466f59c
I20240506 19:56:53.223280 12513 catalog_manager.cc:6554] ReplaceTablet: tablet 260d7ab458794df496012cb80466f59c deleted and replaced by tablet f1809a8a6a9f458bb5d9de9bcf7bd179
I20240506 19:56:53.223009 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 934) took 1195 ms. Trace:
I20240506 19:56:53.220661 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ce7f4ceb9d40009e9c75543b1fc545" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:53.224493 13246 rpcz_store.cc:276] 0506 19:56:52.027087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.028083 (+ 996us) service_pool.cc:225] Handling call
0506 19:56:53.163679 (+1135596us) ts_tablet_manager.cc:1120] Deleting tablet 67e8b8be953a4a5784f2d098fd6364e3
0506 19:56:53.163688 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.167648 (+ 3960us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.221658 (+ 54010us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.222993 (+ 1335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134073}
I20240506 19:56:53.224627 12982 raft_consensus.cc:3037] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.224712 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 260d7ab458794df496012cb80466f59c with delete_type TABLET_DATA_DELETED (replaced by tablet f1809a8a6a9f458bb5d9de9bcf7bd179) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.224905 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 940) took 1133 ms. Trace:
W20240506 19:56:53.225229 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b803ae289344f88b604a2053881d59b consensus state has no leader
I20240506 19:56:53.226037 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0bcbf850d4914899a72f3b786d8b80b0
I20240506 19:56:53.227103 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 260d7ab458794df496012cb80466f59c with delete_type TABLET_DATA_DELETED (replaced by tablet f1809a8a6a9f458bb5d9de9bcf7bd179) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.227138 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bcbf850d4914899a72f3b786d8b80b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.224985 13246 rpcz_store.cc:276] 0506 19:56:52.091131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.091193 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:53.224782 (+1133589us) ts_tablet_manager.cc:1120] Deleting tablet 35198d5786c14de89c9751a0372ba91d
0506 19:56:53.224793 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.224891 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133301}
I20240506 19:56:53.227618 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 942) took 1084 ms. Trace:
I20240506 19:56:53.227665 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bcbf850d4914899a72f3b786d8b80b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.227722 13246 rpcz_store.cc:276] 0506 19:56:52.143171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.143233 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:53.227510 (+1084277us) ts_tablet_manager.cc:1120] Deleting tablet 910d1465df784dfba6d78ff7fd9f2722
0506 19:56:53.227521 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.227604 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084014}
I20240506 19:56:53.224401 13245 raft_consensus.cc:374] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.229934 13245 raft_consensus.cc:729] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.230235 13245 consensus_queue.cc:260] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.231772 13245 ts_tablet_manager.cc:1398] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.048s user 0.000s sys 0.002s
I20240506 19:56:53.232319 13245 tablet_bootstrap.cc:492] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.233067 12982 raft_consensus.cc:2454] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:53.234136 13245 tablet_bootstrap.cc:654] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.236205 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 260d7ab458794df496012cb80466f59c with delete_type TABLET_DATA_DELETED (replaced by tablet f1809a8a6a9f458bb5d9de9bcf7bd179) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.236622 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bcbf850d4914899a72f3b786d8b80b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.237120 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:53.237437 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:53.237751 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:53.238018 12522 catalog_manager.cc:5484] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 2, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 2 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:53.238448 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:53.238713 12522 catalog_manager.cc:5484] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:53.240702 13057 leader_election.cc:302] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.241207 14404 raft_consensus.cc:2781] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:53.241418 14404 raft_consensus.cc:686] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:53.241756 14404 consensus_queue.cc:237] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.251050 12958 tablet_service.cc:1467] Processing CreateTablet for tablet abca438ca98849e5a15b3a26a7eab979 (DEFAULT_TABLE table=default.Table_74ef5084696549ca903c62d3a459320b [id=d0a5c899bb444400a13fd2bdcedb8c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.251432 13245 tablet_bootstrap.cc:492] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.251536 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abca438ca98849e5a15b3a26a7eab979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.251650 13245 ts_tablet_manager.cc:1367] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.005s
I20240506 19:56:53.253170 13242 tablet_replica.cc:330] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:53.253394 13242 raft_consensus.cc:2229] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.253571 13242 raft_consensus.cc:2258] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.254130 13242 ts_tablet_manager.cc:1875] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.256997 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 67e8b8be953a4a5784f2d098fd6364e3 (table default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb [id=7a2c513cd8e74817b4a391c9c7932085]) successfully deleted
W20240506 19:56:53.257316 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 35198d5786c14de89c9751a0372ba91d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35198d5786c14de89c9751a0372ba91d already in progress: creating tablet
W20240506 19:56:53.257540 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 910d1465df784dfba6d78ff7fd9f2722 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 910d1465df784dfba6d78ff7fd9f2722 already in progress: creating tablet
W20240506 19:56:53.257740 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5fe8d4891c484627ac729971ff7aad3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fe8d4891c484627ac729971ff7aad3e already in progress: creating tablet
W20240506 19:56:53.257937 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4f4213fa718b491881f304b4e8f0bf7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f4213fa718b491881f304b4e8f0bf7a already in progress: creating tablet
W20240506 19:56:53.258108 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f424e713b96d43cdac6597c20bd5fc73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f424e713b96d43cdac6597c20bd5fc73 already in progress: creating tablet
W20240506 19:56:53.258280 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet e6b24176d9d941d4a9dfb97e24b91ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6b24176d9d941d4a9dfb97e24b91ab6 already in progress: creating tablet
I20240506 19:56:53.258962 13244 tablet_bootstrap.cc:492] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:53.259860 13245 raft_consensus.cc:348] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.260136 13245 raft_consensus.cc:374] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.260221 13245 raft_consensus.cc:729] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.260486 13245 consensus_queue.cc:260] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.260681 13244 tablet_bootstrap.cc:654] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.261075 13245 ts_tablet_manager.cc:1398] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:53.261444 13245 tablet_bootstrap.cc:492] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.262903 12820 tablet_service.cc:1467] Processing CreateTablet for tablet abca438ca98849e5a15b3a26a7eab979 (DEFAULT_TABLE table=default.Table_74ef5084696549ca903c62d3a459320b [id=d0a5c899bb444400a13fd2bdcedb8c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.263082 13245 tablet_bootstrap.cc:654] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.263281 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abca438ca98849e5a15b3a26a7eab979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.263880 14383 raft_consensus.cc:482] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.264027 14383 raft_consensus.cc:504] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.217197 14402 leader_election.cc:288] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.275142 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 530de4f3f9584ea3b695e3abc08ad560 (DEFAULT_TABLE table=default.Table_b2e9c53dcdb14928b637180cdca5f786 [id=3b6c71a27559456aa2d11f0736cf5dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.275620 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530de4f3f9584ea3b695e3abc08ad560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.278820 13245 tablet_bootstrap.cc:492] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.279032 13245 ts_tablet_manager.cc:1367] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240506 19:56:53.279242 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 467c33f012584d489c2842e3e606f485
I20240506 19:56:53.279971 13245 raft_consensus.cc:348] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.280241 13245 raft_consensus.cc:374] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.280344 13245 raft_consensus.cc:729] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.280365 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 62af97bfc4344c57b3b13ebd01f003be
I20240506 19:56:53.280615 13245 consensus_queue.cc:260] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.281219 13245 ts_tablet_manager.cc:1398] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:53.282588 14383 leader_election.cc:288] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.282714 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 467c33f012584d489c2842e3e606f485 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.282932 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 62af97bfc4344c57b3b13ebd01f003be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:53.283155 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 260d7ab458794df496012cb80466f59c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 260d7ab458794df496012cb80466f59c already in progress: creating tablet
W20240506 19:56:53.283391 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0bcbf850d4914899a72f3b786d8b80b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bcbf850d4914899a72f3b786d8b80b0 already in progress: creating tablet
I20240506 19:56:53.283764 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb357a7eb2147a79cc69d024bd35269" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.283756 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb357a7eb2147a79cc69d024bd35269" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.283874 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 62af97bfc4344c57b3b13ebd01f003be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.283995 13123 raft_consensus.cc:2454] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:53.284441 12922 leader_election.cc:302] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:53.284897 12921 leader_election.cc:341] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.285197 14383 raft_consensus.cc:2781] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.285260 14383 raft_consensus.cc:482] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.285431 14383 raft_consensus.cc:3037] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:53.285627 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 467c33f012584d489c2842e3e606f485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 467c33f012584d489c2842e3e606f485 already in progress: creating tablet
W20240506 19:56:53.285862 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 62af97bfc4344c57b3b13ebd01f003be because the tablet was not found. No further retry: Not found: Tablet not found: 62af97bfc4344c57b3b13ebd01f003be
I20240506 19:56:53.286365 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cf12763862094c8197c09b72b43f4616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.286979 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf12763862094c8197c09b72b43f4616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.287773 14383 raft_consensus.cc:504] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.288328 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_971fe431636342168d3237de7c0a271d" } new_table_name: "default.Table_3f897aef86e1464082e99df8a664489a" modify_external_catalogs: true
I20240506 19:56:53.289268 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb357a7eb2147a79cc69d024bd35269" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:53.289628 13123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb357a7eb2147a79cc69d024bd35269" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:53.289841 13123 raft_consensus.cc:3037] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.290725 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4450908020fd475c8c535415af0a8925" } new_table_name: "default.Table_78979d94cf774d05a800befc0a5b9745" modify_external_catalogs: true
I20240506 19:56:53.292001 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6b08752503f442de813189d8b7dea7b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:53.292495 13123 raft_consensus.cc:2454] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:53.292512 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b08752503f442de813189d8b7dea7b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.293725 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4dcb29fa5425483296ab644d95581803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.294184 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dcb29fa5425483296ab644d95581803 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.294297 13249 tablet_bootstrap.cc:492] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.294458 13249 ts_tablet_manager.cc:1367] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.004s
I20240506 19:56:53.294905 12922 leader_election.cc:302] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:53.295163 14274 raft_consensus.cc:2781] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:53.295284 14274 raft_consensus.cc:686] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:53.295265 13249 raft_consensus.cc:348] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.295516 14274 consensus_queue.cc:237] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.295529 13249 raft_consensus.cc:374] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.296149 13249 raft_consensus.cc:729] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.296356 13249 consensus_queue.cc:260] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:53.296744 12921 leader_election.cc:341] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.296855 13249 ts_tablet_manager.cc:1398] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:53.281544 13245 tablet_bootstrap.cc:492] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.297840 14383 leader_election.cc:288] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.298416 13249 tablet_bootstrap.cc:492] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.301259 13249 tablet_bootstrap.cc:654] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.301481 13245 tablet_bootstrap.cc:654] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.299526 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 467c33f012584d489c2842e3e606f485 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.311547 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 530de4f3f9584ea3b695e3abc08ad560 (DEFAULT_TABLE table=default.Table_b2e9c53dcdb14928b637180cdca5f786 [id=3b6c71a27559456aa2d11f0736cf5dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.312098 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530de4f3f9584ea3b695e3abc08ad560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.317669 13245 tablet_bootstrap.cc:492] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.317930 13245 ts_tablet_manager.cc:1367] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.005s
I20240506 19:56:53.319007 13245 raft_consensus.cc:348] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.319293 13245 raft_consensus.cc:374] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.319371 13245 raft_consensus.cc:729] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:53.319568 12508 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 61310.5us
I20240506 19:56:53.319707 13245 consensus_queue.cc:260] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.320293 13245 ts_tablet_manager.cc:1398] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:53.320660 13245 tablet_bootstrap.cc:492] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:53.321882 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 260d7ab458794df496012cb80466f59c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 260d7ab458794df496012cb80466f59c already in progress: creating tablet
W20240506 19:56:53.322186 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0bcbf850d4914899a72f3b786d8b80b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bcbf850d4914899a72f3b786d8b80b0 already in progress: creating tablet
I20240506 19:56:53.322429 13245 tablet_bootstrap.cc:654] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.323679 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 467c33f012584d489c2842e3e606f485 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.323740 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 62af97bfc4344c57b3b13ebd01f003be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.323793 13096 tablet_service.cc:1467] Processing CreateTablet for tablet abca438ca98849e5a15b3a26a7eab979 (DEFAULT_TABLE table=default.Table_74ef5084696549ca903c62d3a459320b [id=d0a5c899bb444400a13fd2bdcedb8c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.324205 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abca438ca98849e5a15b3a26a7eab979. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.330813 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 467c33f012584d489c2842e3e606f485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 467c33f012584d489c2842e3e606f485 already in progress: creating tablet
W20240506 19:56:53.331141 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 62af97bfc4344c57b3b13ebd01f003be because the tablet was not found. No further retry: Not found: Tablet not found: 62af97bfc4344c57b3b13ebd01f003be
I20240506 19:56:53.333436 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa2d54d5fbf47fe96dfbb9f34b0578c (DEFAULT_TABLE table=default.Table_c30aa0d022c449e8b993bfa4e61fec2e [id=7754f3fa329242a08c91a80f16bebedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.333989 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa2d54d5fbf47fe96dfbb9f34b0578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.336766 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa2d54d5fbf47fe96dfbb9f34b0578c (DEFAULT_TABLE table=default.Table_c30aa0d022c449e8b993bfa4e61fec2e [id=7754f3fa329242a08c91a80f16bebedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.337446 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa2d54d5fbf47fe96dfbb9f34b0578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.355083 13242 ts_tablet_manager.cc:1888] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.356470 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 2d407307e422457eab7e4ada0f909024 (DEFAULT_TABLE table=default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e [id=330c88093b4b4bcfa831c77ce9b44115]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.357331 12958 tablet_service.cc:1467] Processing CreateTablet for tablet edaaead0cf2e4133b8bdec62fb4770a0 (DEFAULT_TABLE table=default.Table_515baa4fc8794cbf904ee3ed951e98b3 [id=cce1e0fef8334ecda5f4cd3d3dfae6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.357748 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaaead0cf2e4133b8bdec62fb4770a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.361732 13249 tablet_bootstrap.cc:492] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.361902 13249 ts_tablet_manager.cc:1367] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.010s
I20240506 19:56:53.361838 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 2d407307e422457eab7e4ada0f909024 (DEFAULT_TABLE table=default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e [id=330c88093b4b4bcfa831c77ce9b44115]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.361840 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa2d54d5fbf47fe96dfbb9f34b0578c (DEFAULT_TABLE table=default.Table_c30aa0d022c449e8b993bfa4e61fec2e [id=7754f3fa329242a08c91a80f16bebedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.362370 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa2d54d5fbf47fe96dfbb9f34b0578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.362658 12952 tablet_service.cc:1467] Processing CreateTablet for tablet bac098377eea4798b7a5516a2a7bb5b0 (DEFAULT_TABLE table=default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.362982 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac098377eea4798b7a5516a2a7bb5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.362922 13249 raft_consensus.cc:348] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.363123 13249 raft_consensus.cc:374] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.363181 13249 raft_consensus.cc:729] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.364315 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 530de4f3f9584ea3b695e3abc08ad560 (DEFAULT_TABLE table=default.Table_b2e9c53dcdb14928b637180cdca5f786 [id=3b6c71a27559456aa2d11f0736cf5dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.364701 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530de4f3f9584ea3b695e3abc08ad560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.364861 12967 tablet_service.cc:1467] Processing CreateTablet for tablet acf5e347bb68466e811259132e3c99b1 (DEFAULT_TABLE table=default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.365190 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf5e347bb68466e811259132e3c99b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.365655 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d407307e422457eab7e4ada0f909024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.363663 13249 consensus_queue.cc:260] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.366793 13249 ts_tablet_manager.cc:1398] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240506 19:56:53.367162 13249 tablet_bootstrap.cc:492] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.367764 14314 raft_consensus.cc:482] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.367871 14314 raft_consensus.cc:504] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.368436 14314 leader_election.cc:288] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.370776 14402 raft_consensus.cc:482] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.371084 14402 raft_consensus.cc:504] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:53.371289 12921 consensus_peers.cc:485] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:53.372064 14402 leader_election.cc:288] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.372270 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2be78cbc8264f6e8d2e05137135161c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.372464 12982 raft_consensus.cc:2454] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:53.372857 13057 leader_election.cc:302] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:53.374172 14402 raft_consensus.cc:482] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.374336 14402 raft_consensus.cc:504] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.375875 13249 tablet_bootstrap.cc:654] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.375993 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452a83470585433e8def7174300d3a7f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:53.376431 13057 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.377055 14404 raft_consensus.cc:2781] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.377164 14404 raft_consensus.cc:482] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.377269 14404 raft_consensus.cc:3037] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.377931 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.378001 13099 tablet_service.cc:1467] Processing CreateTablet for tablet bac098377eea4798b7a5516a2a7bb5b0 (DEFAULT_TABLE table=default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.378414 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac098377eea4798b7a5516a2a7bb5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.378605 13120 raft_consensus.cc:2454] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:53.380040 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b2be78cbc8264f6e8d2e05137135161c
I20240506 19:56:53.380836 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bbde4cb2e81b4723a42d8d0c2006c1a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.381394 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbde4cb2e81b4723a42d8d0c2006c1a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.381762 12922 leader_election.cc:302] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:53.382277 14383 raft_consensus.cc:2781] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.382342 14383 raft_consensus.cc:482] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.382400 14383 raft_consensus.cc:3037] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.382706 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet cf6d0f3900154554b7af9338c95b5122
I20240506 19:56:53.382875 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet b2be78cbc8264f6e8d2e05137135161c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.383246 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 1a6e0126d9854215a0e174a76c0a0391 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:53.383361 14421 tablet_replica.cc:330] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:53.383543 14421 raft_consensus.cc:2229] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.383692 14421 raft_consensus.cc:2258] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.383812 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f08ac26a65b94147960c37766cca1fe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:53.384164 14421 ts_tablet_manager.cc:1875] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:53.384255 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f08ac26a65b94147960c37766cca1fe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.384706 14383 raft_consensus.cc:504] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.386013 14383 leader_election.cc:288] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.386917 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:53.387133 13124 raft_consensus.cc:3037] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.387074 13089 tablet_service.cc:1467] Processing CreateTablet for tablet edaaead0cf2e4133b8bdec62fb4770a0 (DEFAULT_TABLE table=default.Table_515baa4fc8794cbf904ee3ed951e98b3 [id=cce1e0fef8334ecda5f4cd3d3dfae6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.387673 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaaead0cf2e4133b8bdec62fb4770a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.388417 14383 consensus_queue.cc:1035] T 3bb5a12521044102ae55ca953f1496f9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:53.391003 14421 ts_tablet_manager.cc:1888] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.392405 14211 raft_consensus.cc:482] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.392606 14211 raft_consensus.cc:504] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.393285 14211 leader_election.cc:288] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.393456 13124 raft_consensus.cc:2454] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:53.393864 14421 log.cc:1196] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/b2be78cbc8264f6e8d2e05137135161c
I20240506 19:56:53.394268 14421 ts_tablet_manager.cc:1909] T b2be78cbc8264f6e8d2e05137135161c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:53.394824 12922 leader_election.cc:302] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:53.394910 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet b2be78cbc8264f6e8d2e05137135161c (table default.Table_1b37b87a1904426883b754783f6d5c2e [id=93500e72118e41ffac81ba66586212d9]) successfully deleted
I20240506 19:56:53.395272 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6d0f3900154554b7af9338c95b5122 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.395797 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d407307e422457eab7e4ada0f909024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.395895 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997ecba7fbae41178a25bf7af4d152e1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:53.396440 13057 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.396761 14274 raft_consensus.cc:2781] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:53.396925 14274 raft_consensus.cc:686] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:53.397194 14274 consensus_queue.cc:237] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:53.397756 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet cf6d0f3900154554b7af9338c95b5122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf6d0f3900154554b7af9338c95b5122 already in progress: creating tablet
I20240506 19:56:53.399045 14402 leader_election.cc:288] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.402731 14404 raft_consensus.cc:504] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.403515 14383 raft_consensus.cc:482] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.403712 14383 raft_consensus.cc:504] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.405050 14383 leader_election.cc:288] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.409032 13249 tablet_bootstrap.cc:492] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.409175 13249 ts_tablet_manager.cc:1367] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.005s
I20240506 19:56:53.410106 13249 raft_consensus.cc:348] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.410292 13249 raft_consensus.cc:374] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.410349 13249 raft_consensus.cc:729] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.410857 13249 consensus_queue.cc:260] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.411355 13249 ts_tablet_manager.cc:1398] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:53.411625 13249 tablet_bootstrap.cc:492] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.412608 12964 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b7b4c2f924266b7ef2376efc5143d (DEFAULT_TABLE table=default.Table_7c15bd15fd9f442a986827cad7197380 [id=72af3ab851d74256aa8a9a7ad6baaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.413086 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b7b4c2f924266b7ef2376efc5143d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.413172 13249 tablet_bootstrap.cc:654] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.420078 13249 tablet_bootstrap.cc:492] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
W20240506 19:56:53.420109 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57a58ed1c4bd4f80834f9c6f52355334 consensus state has no leader
I20240506 19:56:53.420231 13249 ts_tablet_manager.cc:1367] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240506 19:56:53.420513 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1513820768504912ad5f7b9966ada614" } new_table_name: "default.Table_16507180ad3142ee968513548406bd0b" modify_external_catalogs: true
I20240506 19:56:53.422073 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 79592a4ecacb4324b2278e1b9a75496c from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:53.422660 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_00c419ed22644c6cb9dd58b4b5d5ca39" } new_table_name: "default.Table_c280dff5527d4a95ad135ceba07f8a7f" modify_external_catalogs: true
I20240506 19:56:53.421005 13249 raft_consensus.cc:348] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.423910 13249 raft_consensus.cc:374] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.423980 13249 raft_consensus.cc:729] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.424537 13249 consensus_queue.cc:260] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.424530 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d7cbeb176fef4a638c8709d353a7a395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.425015 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7cbeb176fef4a638c8709d353a7a395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.425063 13249 ts_tablet_manager.cc:1398] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:53.425370 13249 tablet_bootstrap.cc:492] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:53.427433 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adbaf70315ff442e996870d9b6ddcf48 consensus state has no leader
I20240506 19:56:53.428552 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_eb945644f4344376aa5059f1a8223ce9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:53.428746 13249 tablet_bootstrap.cc:654] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:53.429086 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb945644f4344376aa5059f1a8223ce9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.440124 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f9d1e5113608496eb4aad3218eb44a65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.440801 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9d1e5113608496eb4aad3218eb44a65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.444698 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9277b5f5bbf44d878345b1028267ba8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.445428 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9277b5f5bbf44d878345b1028267ba8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.449997 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a2925684b6564128a1301dfe07064f5f" } modify_external_catalogs: true
I20240506 19:56:53.450201 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a2925684b6564128a1301dfe07064f5f" } modify_external_catalogs: true
I20240506 19:56:53.450803 14383 raft_consensus.cc:482] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.450964 14383 raft_consensus.cc:504] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.455987 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Bootstrap replayed 1/1 log segments. Stats: ops{read=584 overwritten=0 applied=581 ignored=0} inserts{seen=535 ignored=0} mutations{seen=657 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240506 19:56:53.457007 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: Bootstrap complete.
I20240506 19:56:53.465898 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 9647ff7be0334c81b75b10f1093e7432 (DEFAULT_TABLE table=default.Table_c2e72287331646f2b30cf2f0e8d0509a [id=d3a119ec22a04657babb58caf69d7604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.466356 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9647ff7be0334c81b75b10f1093e7432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.358719 12817 tablet_service.cc:1467] Processing CreateTablet for tablet bac098377eea4798b7a5516a2a7bb5b0 (DEFAULT_TABLE table=default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.358719 12824 tablet_service.cc:1467] Processing CreateTablet for tablet edaaead0cf2e4133b8bdec62fb4770a0 (DEFAULT_TABLE table=default.Table_515baa4fc8794cbf904ee3ed951e98b3 [id=cce1e0fef8334ecda5f4cd3d3dfae6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.363488 14331 raft_consensus.cc:482] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.466768 14331 raft_consensus.cc:504] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.467175 13242 log.cc:1196] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/e17197adc6ca4593ac3fc494548a6e67
I20240506 19:56:53.467504 14331 leader_election.cc:288] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.467445 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.467578 13242 ts_tablet_manager.cc:1909] T e17197adc6ca4593ac3fc494548a6e67 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:53.467622 14331 raft_consensus.cc:482] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.467725 14331 raft_consensus.cc:504] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.467876 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 800) took 3279 ms. Trace:
I20240506 19:56:53.467875 14383 leader_election.cc:288] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.468209 14331 leader_election.cc:288] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:53.369859 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.467945 13242 rpcz_store.cc:276] 0506 19:56:50.188497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.189152 (+ 655us) service_pool.cc:225] Handling call
0506 19:56:53.212478 (+3023326us) ts_tablet_manager.cc:1120] Deleting tablet e17197adc6ca4593ac3fc494548a6e67
0506 19:56:53.253066 (+ 40588us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.351275 (+ 98209us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.354986 (+ 3711us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.467848 (+112862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3023093}
I20240506 19:56:53.372242 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2be78cbc8264f6e8d2e05137135161c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.375332 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452a83470585433e8def7174300d3a7f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.468523 13242 tablet_replica.cc:330] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:53.385685 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6d0f3900154554b7af9338c95b5122 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.468709 13242 raft_consensus.cc:2229] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.385862 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:53.390686 13244 tablet_bootstrap.cc:492] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:53.394050 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997ecba7fbae41178a25bf7af4d152e1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.468891 13244 ts_tablet_manager.cc:1367] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.210s user 0.001s sys 0.002s
I20240506 19:56:53.399987 12826 tablet_service.cc:1467] Processing CreateTablet for tablet acf5e347bb68466e811259132e3c99b1 (DEFAULT_TABLE table=default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.405571 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04738def56324a368f8182543107e32e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.412595 12827 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b7b4c2f924266b7ef2376efc5143d (DEFAULT_TABLE table=default.Table_7c15bd15fd9f442a986827cad7197380 [id=72af3ab851d74256aa8a9a7ad6baaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.414992 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 9647ff7be0334c81b75b10f1093e7432 (DEFAULT_TABLE table=default.Table_c2e72287331646f2b30cf2f0e8d0509a [id=d3a119ec22a04657babb58caf69d7604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.390851 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet b2be78cbc8264f6e8d2e05137135161c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.466930 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac098377eea4798b7a5516a2a7bb5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.469401 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet e17197adc6ca4593ac3fc494548a6e67 (table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]) successfully deleted
I20240506 19:56:53.469641 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9647ff7be0334c81b75b10f1093e7432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.470860 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaaead0cf2e4133b8bdec62fb4770a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.470192 14199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:53.449580 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 8159ae26e51b423dbe56a9776bd538ad (DEFAULT_TABLE table=default.Table_686d4b3ab7454f2e87e3fab0797c3cee [id=bae798c3d8fb4998977e22133b0c6d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.473402 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf5e347bb68466e811259132e3c99b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.473553 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8159ae26e51b423dbe56a9776bd538ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.477407 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b7b4c2f924266b7ef2376efc5143d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.479866 12921 leader_election.cc:341] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.479948 12844 raft_consensus.cc:3037] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 2
W20240506 19:56:53.480106 12921 leader_election.cc:341] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.480406 12921 leader_election.cc:341] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.480652 12921 leader_election.cc:341] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.481981 12783 leader_election.cc:341] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:53.482419 14331 raft_consensus.cc:482] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.482535 14331 raft_consensus.cc:504] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.482640 13244 raft_consensus.cc:348] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.482797 13244 raft_consensus.cc:374] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.482843 13244 raft_consensus.cc:729] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.483037 13244 consensus_queue.cc:260] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.483197 14331 leader_election.cc:288] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.483266 12844 raft_consensus.cc:1261] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 2 FOLLOWER]: Refusing update from remote peer 059a5c4a3f7e49ef95af4b8341702745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240506 19:56:53.483525 14199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f390a786034cddbff757779037f861, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.483527 13244 ts_tablet_manager.cc:1398] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.015s user 0.000s sys 0.000s
I20240506 19:56:53.484463 14199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 581, Last appended: 1.584, Last appended by leader: 584, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
W20240506 19:56:53.485102 12783 leader_election.cc:341] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2417a535329341d8bb9f8218770f890d
I20240506 19:56:53.488361 13244 tablet_bootstrap.cc:492] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:53.488811 12781 leader_election.cc:341] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2417a535329341d8bb9f8218770f890d
I20240506 19:56:53.488910 12781 leader_election.cc:302] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:53.489413 14331 raft_consensus.cc:2726] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.490015 13244 tablet_bootstrap.cc:654] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.491747 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 8159ae26e51b423dbe56a9776bd538ad (DEFAULT_TABLE table=default.Table_686d4b3ab7454f2e87e3fab0797c3cee [id=bae798c3d8fb4998977e22133b0c6d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.492173 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8159ae26e51b423dbe56a9776bd538ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.492427 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.493350 12821 tablet_service.cc:1467] Processing CreateTablet for tablet ede01d0bed034c16bb580124bd64d241 (DEFAULT_TABLE table=default.Table_1403b3f7ba884b308e7346ed7eaa06c6 [id=36c18887a18442ab9f91deb76c49af64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.493744 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede01d0bed034c16bb580124bd64d241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.493865 12820 tablet_service.cc:1467] Processing CreateTablet for tablet c97f87e32bd34cdb91d3f2edd1827266 (DEFAULT_TABLE table=default.Table_90043eda47664c8fa37013b6730e5fb0 [id=62a40b2784264beb924df20c9798bc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.494731 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97f87e32bd34cdb91d3f2edd1827266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.498445 12825 tablet_service.cc:1467] Processing CreateTablet for tablet cae859ab30064c969dbf040da2c14524 (DEFAULT_TABLE table=default.Table_bc52ab9a06b149e999bf34c308e24839 [id=d8af7c79f3994f49b9bd7c11eacd65e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.499068 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae859ab30064c969dbf040da2c14524. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.500545 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 582725a8a48546a68937f7d11a1873a4 consensus state has no leader
I20240506 19:56:53.500927 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3e8858530f1441ffa975a8576f0ffd5b" } new_table_name: "default.Table_b24141a2526d4993b9210b8cc83dd50c" modify_external_catalogs: true
I20240506 19:56:53.502658 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fb84f06ac8d34f659ca376315100395f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.503171 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb84f06ac8d34f659ca376315100395f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.504016 14383 consensus_queue.cc:1035] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:53.504544 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aee896bd7f14475a8bba1a5228d590da consensus state has no leader
I20240506 19:56:53.508827 13242 raft_consensus.cc:2258] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.373751 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 2d407307e422457eab7e4ada0f909024 (DEFAULT_TABLE table=default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e [id=330c88093b4b4bcfa831c77ce9b44115]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.509723 13242 ts_tablet_manager.cc:1875] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.515311 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d407307e422457eab7e4ada0f909024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.524107 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f11c6836fece4a3fb0a20a2ad29761a3" } modify_external_catalogs: true
I20240506 19:56:53.524363 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f11c6836fece4a3fb0a20a2ad29761a3" } modify_external_catalogs: true
I20240506 19:56:53.528293 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_73585d09b2764941be3d724930ffd097" } modify_external_catalogs: true
I20240506 19:56:53.528513 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_73585d09b2764941be3d724930ffd097" } modify_external_catalogs: true
W20240506 19:56:53.529546 12781 leader_election.cc:341] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:53.529692 12781 leader_election.cc:302] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:53.530153 14331 raft_consensus.cc:2726] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.531692 12967 tablet_service.cc:1467] Processing CreateTablet for tablet ede01d0bed034c16bb580124bd64d241 (DEFAULT_TABLE table=default.Table_1403b3f7ba884b308e7346ed7eaa06c6 [id=36c18887a18442ab9f91deb76c49af64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.531827 12952 tablet_service.cc:1467] Processing CreateTablet for tablet c97f87e32bd34cdb91d3f2edd1827266 (DEFAULT_TABLE table=default.Table_90043eda47664c8fa37013b6730e5fb0 [id=62a40b2784264beb924df20c9798bc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.532398 12958 tablet_service.cc:1467] Processing CreateTablet for tablet cae859ab30064c969dbf040da2c14524 (DEFAULT_TABLE table=default.Table_bc52ab9a06b149e999bf34c308e24839 [id=d8af7c79f3994f49b9bd7c11eacd65e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.532588 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede01d0bed034c16bb580124bd64d241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.532752 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae859ab30064c969dbf040da2c14524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.532948 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 01f0def08b17403180b9b876c26428f7 (DEFAULT_TABLE table=default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.533329 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f0def08b17403180b9b876c26428f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.405819 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04738def56324a368f8182543107e32e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.407877 13245 tablet_bootstrap.cc:492] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.410879 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6d0f3900154554b7af9338c95b5122 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.538408 13124 raft_consensus.cc:2454] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:53.538408 13245 ts_tablet_manager.cc:1367] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.218s user 0.006s sys 0.001s
I20240506 19:56:53.539176 12922 leader_election.cc:302] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.539543 12825 tablet_service.cc:1467] Processing CreateTablet for tablet aee896bd7f14475a8bba1a5228d590da (DEFAULT_TABLE table=default.Table_c280dff5527d4a95ad135ceba07f8a7f [id=ae97ea5c98db4ea4baf292ee3eef4136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.539603 14404 leader_election.cc:288] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.539674 13245 raft_consensus.cc:348] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.539880 13245 raft_consensus.cc:374] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.539978 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee896bd7f14475a8bba1a5228d590da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.539994 13245 raft_consensus.cc:729] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.540244 14274 raft_consensus.cc:2781] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.540320 14274 raft_consensus.cc:482] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.540212 13245 consensus_queue.cc:260] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.540390 14274 raft_consensus.cc:3037] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.540750 13245 ts_tablet_manager.cc:1398] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:53.541028 13245 tablet_bootstrap.cc:492] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.542632 13245 tablet_bootstrap.cc:654] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.543382 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97f87e32bd34cdb91d3f2edd1827266. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.543354 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet cf6d0f3900154554b7af9338c95b5122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf6d0f3900154554b7af9338c95b5122 already in progress: creating tablet
I20240506 19:56:53.544198 13093 tablet_service.cc:1467] Processing CreateTablet for tablet aee896bd7f14475a8bba1a5228d590da (DEFAULT_TABLE table=default.Table_c280dff5527d4a95ad135ceba07f8a7f [id=ae97ea5c98db4ea4baf292ee3eef4136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.544385 13244 tablet_bootstrap.cc:492] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:56:53.544425 13058 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.544502 13244 ts_tablet_manager.cc:1367] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20240506 19:56:53.544636 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee896bd7f14475a8bba1a5228d590da. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.545064 13058 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.545172 13058 leader_election.cc:302] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.545217 13244 raft_consensus.cc:348] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.545418 13244 raft_consensus.cc:374] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.545465 13244 raft_consensus.cc:729] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.545646 13244 consensus_queue.cc:260] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.545944 14398 raft_consensus.cc:2726] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.546155 13244 ts_tablet_manager.cc:1398] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:53.546396 13244 tablet_bootstrap.cc:492] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:53.547812 13244 tablet_bootstrap.cc:654] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:53.548942 13058 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.549064 13058 leader_election.cc:302] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.549468 14398 raft_consensus.cc:2726] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.549986 13086 tablet_service.cc:1467] Processing CreateTablet for tablet cae859ab30064c969dbf040da2c14524 (DEFAULT_TABLE table=default.Table_bc52ab9a06b149e999bf34c308e24839 [id=d8af7c79f3994f49b9bd7c11eacd65e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.550109 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2be78cbc8264f6e8d2e05137135161c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:53.550370 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae859ab30064c969dbf040da2c14524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.411070 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet b2be78cbc8264f6e8d2e05137135161c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.413956 13099 tablet_service.cc:1467] Processing CreateTablet for tablet acf5e347bb68466e811259132e3c99b1 (DEFAULT_TABLE table=default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.555120 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf5e347bb68466e811259132e3c99b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.453998 13090 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b7b4c2f924266b7ef2376efc5143d (DEFAULT_TABLE table=default.Table_7c15bd15fd9f442a986827cad7197380 [id=72af3ab851d74256aa8a9a7ad6baaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.555853 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b7b4c2f924266b7ef2376efc5143d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.556581 14274 raft_consensus.cc:504] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.556845 12962 tablet_service.cc:1467] Processing CreateTablet for tablet aee896bd7f14475a8bba1a5228d590da (DEFAULT_TABLE table=default.Table_c280dff5527d4a95ad135ceba07f8a7f [id=ae97ea5c98db4ea4baf292ee3eef4136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.557193 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee896bd7f14475a8bba1a5228d590da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.557219 14274 leader_election.cc:288] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.558074 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 3743fc7f36a74dd9b7c01a6bfb011c9d
W20240506 19:56:53.558804 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cade6b1c5a034b23aa33b2349e1dab95 consensus state has no leader
I20240506 19:56:53.559880 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04738def56324a368f8182543107e32e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:53.466651 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.560303 13119 raft_consensus.cc:2454] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:53.469784 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce272e5b5f6483c9d031dff1104401b" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.498189 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 9647ff7be0334c81b75b10f1093e7432 (DEFAULT_TABLE table=default.Table_c2e72287331646f2b30cf2f0e8d0509a [id=d3a119ec22a04657babb58caf69d7604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.498621 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 8159ae26e51b423dbe56a9776bd538ad (DEFAULT_TABLE table=default.Table_686d4b3ab7454f2e87e3fab0797c3cee [id=bae798c3d8fb4998977e22133b0c6d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.498934 13101 tablet_service.cc:1467] Processing CreateTablet for tablet c97f87e32bd34cdb91d3f2edd1827266 (DEFAULT_TABLE table=default.Table_90043eda47664c8fa37013b6730e5fb0 [id=62a40b2784264beb924df20c9798bc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.499224 13097 tablet_service.cc:1467] Processing CreateTablet for tablet ede01d0bed034c16bb580124bd64d241 (DEFAULT_TABLE table=default.Table_1403b3f7ba884b308e7346ed7eaa06c6 [id=36c18887a18442ab9f91deb76c49af64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.561280 14331 raft_consensus.cc:482] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.561447 14331 raft_consensus.cc:504] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.561487 12783 leader_election.cc:302] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:53.562173 14331 leader_election.cc:288] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:53.562783 12921 leader_election.cc:341] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.562852 12514 catalog_manager.cc:5484] T 07c7c7b2da9e403f8241a512e210c4d9 P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:53.563318 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:53.563565 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:53.563763 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.563828 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:53.564015 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet dcb357a7eb2147a79cc69d024bd35269 (table default.Table_b35bb26efd3c4ca08971e6c1185a6c0b [id=b6aeaf0e19d746dab4a37005dc753d65]). Expected version 1 got 0
I20240506 19:56:53.564110 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04738def56324a368f8182543107e32e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:53.564198 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:53.512876 14402 raft_consensus.cc:482] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.555428 13246 tablet_replica.cc:330] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.564357 13121 raft_consensus.cc:3037] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.564364 12514 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:53.564571 13246 raft_consensus.cc:2229] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.564741 13246 raft_consensus.cc:2258] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.565340 13246 ts_tablet_manager.cc:1875] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.565773 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9647ff7be0334c81b75b10f1093e7432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.566155 12513 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:53.566957 13121 raft_consensus.cc:2454] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:53.567569 12922 leader_election.cc:302] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:53.569000 12783 leader_election.cc:341] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b2a5a5e193f245de96cc82986e292f05
I20240506 19:56:53.569430 12922 leader_election.cc:302] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:53.570399 14402 raft_consensus.cc:504] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.571204 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad527c3f2a4b53affa928e149db1df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.571878 13246 ts_tablet_manager.cc:1888] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.572021 13246 log.cc:1196] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/b2be78cbc8264f6e8d2e05137135161c
I20240506 19:56:53.572352 13246 ts_tablet_manager.cc:1909] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.573510 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8159ae26e51b423dbe56a9776bd538ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.431983 14401 raft_consensus.cc:482] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.574003 14401 raft_consensus.cc:504] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.574347 14274 raft_consensus.cc:2781] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:53.574437 14274 raft_consensus.cc:482] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:53.574497 14274 raft_consensus.cc:3037] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.576965 14274 raft_consensus.cc:504] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.577792 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97f87e32bd34cdb91d3f2edd1827266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.578496 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 07a24a78936e4da68a4af40601b98e96 (DEFAULT_TABLE table=default.Table_99bde890b8c14fe2b57afaa1b6f6ed45 [id=38092479ce4249f5aed0ab679d4374ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.578871 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a24a78936e4da68a4af40601b98e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.581050 14274 leader_election.cc:288] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.581221 14274 raft_consensus.cc:2781] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:53.581351 14274 raft_consensus.cc:686] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:53.581593 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede01d0bed034c16bb580124bd64d241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.581513 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:53.582032 12921 leader_election.cc:341] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.582576 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 7525faf76e174113aa00d5b53aa6ea22 (DEFAULT_TABLE table=default.Table_fdebb50309c74888b6a893ac37f9cb9c [id=5d6532bd90404984afd9c8d0aa099990]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.582618 12827 tablet_service.cc:1467] Processing CreateTablet for tablet abdd4edb2c454681a65e363c88941f58 (DEFAULT_TABLE table=default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.582762 13090 tablet_service.cc:1467] Processing CreateTablet for tablet abdd4edb2c454681a65e363c88941f58 (DEFAULT_TABLE table=default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.583024 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7525faf76e174113aa00d5b53aa6ea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.582993 12813 tablet_service.cc:1467] Processing CreateTablet for tablet e830e166e30d46edb77a2f5a30e4dac6 (DEFAULT_TABLE table=default.Table_6cc42687d6c545068a39c096f2932dcd [id=64268ea18e3b4fbb9037218bcca5bbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.583122 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd4edb2c454681a65e363c88941f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.583289 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e830e166e30d46edb77a2f5a30e4dac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.589313 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet b2be78cbc8264f6e8d2e05137135161c (table default.Table_1b37b87a1904426883b754783f6d5c2e [id=93500e72118e41ffac81ba66586212d9]) successfully deleted
W20240506 19:56:53.589911 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3743fc7f36a74dd9b7c01a6bfb011c9d already in progress: creating tablet
I20240506 19:56:53.590263 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 01f0def08b17403180b9b876c26428f7 (DEFAULT_TABLE table=default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.590960 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f0def08b17403180b9b876c26428f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.581595 14274 consensus_queue.cc:237] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.591321 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd4edb2c454681a65e363c88941f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.596098 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3bd92908f341d99f8756a654799019 (DEFAULT_TABLE table=default.Table_51d50ac0d42d4388a250ec5fd84128a8 [id=0f8c1a749dd94feb907921bc2524dbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.596597 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a3bd92908f341d99f8756a654799019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.600433 12810 tablet_service.cc:1467] Processing CreateTablet for tablet f08494422d044ce9abcf0d56aca64279 (DEFAULT_TABLE table=default.Table_eb4ba3a6295e43989a12e36aa7dcc41a [id=bc157254562147ae9e5ffc52a438affa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.601130 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.602156 13249 tablet_bootstrap.cc:492] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:53.602161 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f1a792bf9b774b8f8cc5491605f504ac
I20240506 19:56:53.602283 13249 ts_tablet_manager.cc:1367] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.177s user 0.006s sys 0.013s
I20240506 19:56:53.602422 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 01f0def08b17403180b9b876c26428f7 (DEFAULT_TABLE table=default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.602869 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f0def08b17403180b9b876c26428f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.602972 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 8bd1b51c240a4b1aad65839679f7bba7
I20240506 19:56:53.604562 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08494422d044ce9abcf0d56aca64279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.604637 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a792bf9b774b8f8cc5491605f504ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.608345 13244 tablet_bootstrap.cc:492] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:53.608476 13244 ts_tablet_manager.cc:1367] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240506 19:56:53.608474 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 07a24a78936e4da68a4af40601b98e96 (DEFAULT_TABLE table=default.Table_99bde890b8c14fe2b57afaa1b6f6ed45 [id=38092479ce4249f5aed0ab679d4374ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.608901 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a24a78936e4da68a4af40601b98e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.609159 13244 raft_consensus.cc:348] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.609385 13244 raft_consensus.cc:374] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.609436 13244 raft_consensus.cc:729] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.609548 12953 tablet_service.cc:1467] Processing CreateTablet for tablet f08494422d044ce9abcf0d56aca64279 (DEFAULT_TABLE table=default.Table_eb4ba3a6295e43989a12e36aa7dcc41a [id=bc157254562147ae9e5ffc52a438affa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.609614 13244 consensus_queue.cc:260] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.609911 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08494422d044ce9abcf0d56aca64279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.610105 13244 ts_tablet_manager.cc:1398] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:53.610363 13244 tablet_bootstrap.cc:492] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:53.611974 13244 tablet_bootstrap.cc:654] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.612901 12960 tablet_service.cc:1467] Processing CreateTablet for tablet abdd4edb2c454681a65e363c88941f58 (DEFAULT_TABLE table=default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.613440 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 392e4e4fa399443cb23b5641ff4c6c39 (DEFAULT_TABLE table=default.Table_c47cda70aad1490e8d9668e7240b0673 [id=8e65e9a5a18a4dbfb6a1b4ebfb76b166]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.613549 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3bd92908f341d99f8756a654799019 (DEFAULT_TABLE table=default.Table_51d50ac0d42d4388a250ec5fd84128a8 [id=0f8c1a749dd94feb907921bc2524dbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.613808 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392e4e4fa399443cb23b5641ff4c6c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.613266 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd4edb2c454681a65e363c88941f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.614972 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd1b51c240a4b1aad65839679f7bba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.592571 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3bd92908f341d99f8756a654799019 (DEFAULT_TABLE table=default.Table_51d50ac0d42d4388a250ec5fd84128a8 [id=0f8c1a749dd94feb907921bc2524dbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.616735 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a3bd92908f341d99f8756a654799019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.617024 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a3bd92908f341d99f8756a654799019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.592888 13097 tablet_service.cc:1467] Processing CreateTablet for tablet f08494422d044ce9abcf0d56aca64279 (DEFAULT_TABLE table=default.Table_eb4ba3a6295e43989a12e36aa7dcc41a [id=bc157254562147ae9e5ffc52a438affa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.620533 14398 raft_consensus.cc:482] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.620708 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08494422d044ce9abcf0d56aca64279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.620759 14398 raft_consensus.cc:504] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.621443 14398 leader_election.cc:288] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.621627 13242 ts_tablet_manager.cc:1888] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.621747 13242 log.cc:1196] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1372be71e1d141ed89e58cdb806431a4
I20240506 19:56:53.621811 14398 raft_consensus.cc:482] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.621933 14398 raft_consensus.cc:504] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.622043 13242 ts_tablet_manager.cc:1909] T 1372be71e1d141ed89e58cdb806431a4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:53.622490 14398 leader_election.cc:288] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.593173 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7525faf76e174113aa00d5b53aa6ea22 (DEFAULT_TABLE table=default.Table_fdebb50309c74888b6a893ac37f9cb9c [id=5d6532bd90404984afd9c8d0aa099990]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.623721 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7525faf76e174113aa00d5b53aa6ea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.594497 14402 leader_election.cc:288] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.606050 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a792bf9b774b8f8cc5491605f504ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.607604 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd1b51c240a4b1aad65839679f7bba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.636153 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 07a24a78936e4da68a4af40601b98e96 (DEFAULT_TABLE table=default.Table_99bde890b8c14fe2b57afaa1b6f6ed45 [id=38092479ce4249f5aed0ab679d4374ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.638445 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a24a78936e4da68a4af40601b98e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.648293 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:53.648861 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:53.649233 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:53.649518 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 9d0d827d732e4261a7a8edb549c1c8e3 (table default.Table_9eea3fa543d74708aed9bf6f5e378d89 [id=25076b7bd89a4c3bbb205acaab09690a]). Expected version 1 got 0
I20240506 19:56:53.649849 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet dcb357a7eb2147a79cc69d024bd35269 (table default.Table_b35bb26efd3c4ca08971e6c1185a6c0b [id=b6aeaf0e19d746dab4a37005dc753d65]). Expected version 1 got 0
I20240506 19:56:53.650166 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 673dc7c83f8748a09c41d51b42214aaf (table default.Table_f44e310a873e4dd28f57ab5ff4e28050 [id=59da284ab7244c98b6627a0263be2c64]). Expected version 2 got 0
I20240506 19:56:53.650403 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:53.650712 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f75467ef60504763a67fc10e8639716c (table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]). Expected version 1 got 0
I20240506 19:56:53.651010 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:53.651479 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]). Expected version 2 got 0
I20240506 19:56:53.651540 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3d13322b4a4a2db3b6d425679f68ed with delete_type TABLET_DATA_DELETED (replaced by tablet a03f574aec984b83ab596e92a6f323fc) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.651887 12513 catalog_manager.cc:5484] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195). New cstate: current_term: 1 leader_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:53.652979 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f8ce7f4ceb9d40009e9c75543b1fc545 (table default.Table_9068971de53c4d30bbe309692dd907f3 [id=7d2da7abfa3e48f79147c8a16820ebcb]). Expected version 1 got 0
W20240506 19:56:53.653105 13058 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.655001 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4e97d10ae54e558920d116d8fa0ba5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.655562 12514 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.195:39651 (request call id 165) took 1260 ms (client timeout 1000 ms). Trace:
W20240506 19:56:53.655673 12514 rpcz_store.cc:269] 0506 19:56:52.394919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.401771 (+ 6852us) service_pool.cc:225] Handling call
0506 19:56:53.655527 (+1253756us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:53.568293 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:53.568417 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:53.568424 (+ 7us) write_op.cc:435] Acquired schema lock
0506 19:56:53.568426 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:53.568539 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:53.568556 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:53.568565 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:53.568640 (+ 75us) tablet.cc:699] Row locks acquired
0506 19:56:53.568642 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:53.568683 (+ 41us) write_op.cc:270] Start()
0506 19:56:53.568721 (+ 38us) write_op.cc:276] Timestamp: P: 1715025413568714 usec, L: 0
0506 19:56:53.568725 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:53.568935 (+ 210us) log.cc:842] Serialized 533 byte log entry
0506 19:56:53.641816 (+ 72881us) op_driver.cc:444] REPLICATION: finished
0506 19:56:53.641997 (+ 181us) write_op.cc:301] APPLY: starting
0506 19:56:53.642019 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:53.642082 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:53.642123 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:53.642197 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:53.642244 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:53.642254 (+ 10us) write_op.cc:312] APPLY: finished
0506 19:56:53.642394 (+ 140us) log.cc:842] Serialized 23 byte log entry
0506 19:56:53.642471 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:53.642511 (+ 40us) write_op.cc:454] Released schema lock
0506 19:56:53.642520 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:53.642585 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:53.642590 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":123,"apply.run_cpu_time_us":652,"apply.run_wall_time_us":652,"num_ops":1,"prepare.queue_time_us":3225,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":1102,"replication_time_us":73048}]]}
I20240506 19:56:53.657740 12522 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741] alter complete (version 3)
I20240506 19:56:53.658963 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116df9bd6ab04a30ba09861e765d7267" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.660511 12522 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.194:59149 (request call id 133) took 1035 ms (client timeout 1000 ms). Trace:
I20240506 19:56:53.660614 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673dc7c83f8748a09c41d51b42214aaf" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.660640 12522 rpcz_store.cc:269] 0506 19:56:52.625377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.666784 (+ 41407us) service_pool.cc:225] Handling call
0506 19:56:53.660490 (+993706us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:53.495474 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:53.495574 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:53.495581 (+ 7us) write_op.cc:435] Acquired schema lock
0506 19:56:53.495584 (+ 3us) tablet.cc:660] Decoding operations
0506 19:56:53.495717 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:53.495729 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:53.495735 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:53.495847 (+ 112us) tablet.cc:699] Row locks acquired
0506 19:56:53.495849 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:53.495896 (+ 47us) write_op.cc:270] Start()
0506 19:56:53.495930 (+ 34us) write_op.cc:276] Timestamp: P: 1715025413495900 usec, L: 0
0506 19:56:53.495934 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:53.496120 (+ 186us) log.cc:842] Serialized 913 byte log entry
0506 19:56:53.553857 (+ 57737us) op_driver.cc:444] REPLICATION: finished
0506 19:56:53.555507 (+ 1650us) write_op.cc:301] APPLY: starting
0506 19:56:53.555519 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:53.555547 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:53.555566 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:53.555608 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:53.555630 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:53.555636 (+ 6us) write_op.cc:312] APPLY: finished
0506 19:56:53.555677 (+ 41us) log.cc:842] Serialized 29 byte log entry
0506 19:56:53.555711 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:53.555723 (+ 12us) write_op.cc:454] Released schema lock
0506 19:56:53.555727 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:53.555766 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:53.555770 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 19:56:53.569430 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:53.569638 (+ 208us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:53.569644 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:53.569646 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:53.569730 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:53.569740 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:53.569744 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:53.569794 (+ 50us) tablet.cc:699] Row locks acquired
0506 19:56:53.569796 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:53.569837 (+ 41us) write_op.cc:270] Start()
0506 19:56:53.569875 (+ 38us) write_op.cc:276] Timestamp: P: 1715025413569861 usec, L: 0
0506 19:56:53.569878 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:53.570022 (+ 144us) log.cc:842] Serialized 356 byte log entry
0506 19:56:53.641890 (+ 71868us) op_driver.cc:444] REPLICATION: finished
0506 19:56:53.642686 (+ 796us) write_op.cc:301] APPLY: starting
0506 19:56:53.642701 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:53.642736 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:53.642766 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:53.642817 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:53.642851 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:53.642860 (+ 9us) write_op.cc:312] APPLY: finished
0506 19:56:53.642917 (+ 57us) log.cc:842] Serialized 23 byte log entry
0506 19:56:53.642957 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:53.642976 (+ 19us) write_op.cc:454] Released schema lock
0506 19:56:53.642983 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:53.643035 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:53.643042 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":1622,"apply.run_cpu_time_us":308,"apply.run_wall_time_us":308,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":1095,"replication_time_us":57907}],["op",{"apply.queue_time_us":758,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":1,"prepare.queue_time_us":1184,"prepare.run_cpu_time_us":555,"prepare.run_wall_time_us":953,"replication_time_us":71994}]]}
W20240506 19:56:53.661507 13058 leader_election.cc:341] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.665014 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd1b51c240a4b1aad65839679f7bba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.665006 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 392e4e4fa399443cb23b5641ff4c6c39 (DEFAULT_TABLE table=default.Table_c47cda70aad1490e8d9668e7240b0673 [id=8e65e9a5a18a4dbfb6a1b4ebfb76b166]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.665370 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.665448 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392e4e4fa399443cb23b5641ff4c6c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.665720 14421 tablet_replica.cc:330] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:53.665910 12948 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a792bf9b774b8f8cc5491605f504ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.665946 14421 raft_consensus.cc:2229] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.666086 14421 raft_consensus.cc:2258] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.666589 14421 ts_tablet_manager.cc:1875] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.666960 12522 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_515baa4fc8794cbf904ee3ed951e98b3" } modify_external_catalogs: true
I20240506 19:56:53.667179 12522 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_515baa4fc8794cbf904ee3ed951e98b3" } modify_external_catalogs: true
I20240506 19:56:53.665251 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.670506 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d7cbeb176fef4a638c8709d353a7a395" } modify_external_catalogs: true
I20240506 19:56:53.670739 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d7cbeb176fef4a638c8709d353a7a395" } modify_external_catalogs: true
I20240506 19:56:53.673506 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 8eb9296bb1104b4cbe279eecd3683796 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:53.674410 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_825048dc8d55425b9dc8c3787acd53ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.674993 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_825048dc8d55425b9dc8c3787acd53ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.678978 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 766d77bb818e429fb9b745e99adea585 (DEFAULT_TABLE table=default.Table_2d668c000c934cf7906b877bc0938038 [id=c87e723310b64d9b8021bce63b144556]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.679643 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 62af97bfc4344c57b3b13ebd01f003be (DEFAULT_TABLE table=default.Table_6f73fec327d24cf7bfabb15d38171100 [id=f3ae423b282a4280a5aedc772a3725b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.679793 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 7525faf76e174113aa00d5b53aa6ea22 (DEFAULT_TABLE table=default.Table_fdebb50309c74888b6a893ac37f9cb9c [id=5d6532bd90404984afd9c8d0aa099990]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.680126 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_54845e03ef4e4cc7b472aeee0c2b4c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:53.680303 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7525faf76e174113aa00d5b53aa6ea22. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.680632 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54845e03ef4e4cc7b472aeee0c2b4c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.681485 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.688388 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766d77bb818e429fb9b745e99adea585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.691637 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_69b33a179131461abb0c8a0c13a419dd" } new_table_name: "default.Table_d2889007e246431bb796c67ca72db0aa" modify_external_catalogs: true
W20240506 19:56:53.693037 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3743fc7f36a74dd9b7c01a6bfb011c9d already in progress: creating tablet
W20240506 19:56:53.693583 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8bd1b51c240a4b1aad65839679f7bba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bd1b51c240a4b1aad65839679f7bba7 already in progress: creating tablet
I20240506 19:56:53.694494 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.694836 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_550853d348524a21be958c7e260c141b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.695502 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_550853d348524a21be958c7e260c141b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.695494 14401 leader_election.cc:288] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.696439 13245 tablet_bootstrap.cc:492] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.696652 13245 ts_tablet_manager.cc:1367] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.156s user 0.016s sys 0.008s
I20240506 19:56:53.697611 14401 raft_consensus.cc:482] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.697870 14401 raft_consensus.cc:504] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.698282 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62af97bfc4344c57b3b13ebd01f003be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.702764 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 904f11fe9eac4fcea9832218a4bed98b (DEFAULT_TABLE table=default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.702886 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d56b1e730d4c4eee9b775d01f57057c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:53.703306 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904f11fe9eac4fcea9832218a4bed98b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.703451 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d56b1e730d4c4eee9b775d01f57057c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.705777 14401 leader_election.cc:288] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.706252 14440 tablet.cc:1722] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:53.707139 14404 raft_consensus.cc:482] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.697432 13245 raft_consensus.cc:348] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.707567 13245 raft_consensus.cc:374] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.707648 13245 raft_consensus.cc:729] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.707859 13245 consensus_queue.cc:260] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.707680 14404 raft_consensus.cc:504] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.708400 13245 ts_tablet_manager.cc:1398] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240506 19:56:53.709152 14404 leader_election.cc:288] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.709712 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_76b20d55cba5431aa5b3b38c9eca1e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:53.710443 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76b20d55cba5431aa5b3b38c9eca1e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:53.713879 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_94602d1502334dcb977668d7d42eebec" } new_table_name: "default.Table_fd059b81c3d045c48dbba6b7f1f2e31d" modify_external_catalogs: true
I20240506 19:56:53.714934 13245 tablet_bootstrap.cc:492] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.723193 13245 tablet_bootstrap.cc:654] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:53.723795 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 801) took 3479 ms. Trace:
I20240506 19:56:53.723932 13242 rpcz_store.cc:276] 0506 19:56:50.244797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.244851 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:53.468482 (+3223631us) ts_tablet_manager.cc:1120] Deleting tablet 1372be71e1d141ed89e58cdb806431a4
0506 19:56:53.468490 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.519166 (+ 50676us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.621518 (+102352us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.723762 (+102244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3223353}
I20240506 19:56:53.724851 13242 tablet_replica.cc:330] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:53.725116 13242 raft_consensus.cc:2229] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.725334 13242 raft_consensus.cc:2258] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.725993 13242 ts_tablet_manager.cc:1875] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.727546 12824 tablet_service.cc:1467] Processing CreateTablet for tablet b904072d89c6486fa3ca1557220f0b52 (DEFAULT_TABLE table=default.Table_7881939105214872a57c956985a14360 [id=c91b5a199c00499186aabe5137b4f59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.727988 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 912fa5b484e74572861f58ce78b1572e (DEFAULT_TABLE table=default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.731652 12964 tablet_service.cc:1467] Processing CreateTablet for tablet e830e166e30d46edb77a2f5a30e4dac6 (DEFAULT_TABLE table=default.Table_6cc42687d6c545068a39c096f2932dcd [id=64268ea18e3b4fbb9037218bcca5bbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.731655 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 62af97bfc4344c57b3b13ebd01f003be (DEFAULT_TABLE table=default.Table_6f73fec327d24cf7bfabb15d38171100 [id=f3ae423b282a4280a5aedc772a3725b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.732136 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62af97bfc4344c57b3b13ebd01f003be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.732357 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e830e166e30d46edb77a2f5a30e4dac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.734690 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 766d77bb818e429fb9b745e99adea585 (DEFAULT_TABLE table=default.Table_2d668c000c934cf7906b877bc0938038 [id=c87e723310b64d9b8021bce63b144556]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.735109 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766d77bb818e429fb9b745e99adea585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.727994 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b904072d89c6486fa3ca1557220f0b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.739145 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa5b484e74572861f58ce78b1572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.745226 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:53.746315 12827 tablet_service.cc:1467] Processing CreateTablet for tablet c09e90928a2442d9bb364e10bb3e405a (DEFAULT_TABLE table=default.Table_d59316fe0b304bd9b03da5f95408a1c6 [id=ac0916562c14439e91b1117812475f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.746829 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09e90928a2442d9bb364e10bb3e405a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.756932 13058 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.757588 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:53.758214 13245 tablet_bootstrap.cc:492] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.758540 13245 ts_tablet_manager.cc:1367] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.001s
I20240506 19:56:53.759255 13245 raft_consensus.cc:348] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.759454 13245 raft_consensus.cc:374] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.759526 13245 raft_consensus.cc:729] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.759734 13245 consensus_queue.cc:260] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:53.759883 13058 leader_election.cc:341] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.764047 13245 ts_tablet_manager.cc:1398] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:53.764393 13245 tablet_bootstrap.cc:492] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.768136 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dafb661b2443f5afadc6816868bac8" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.768203 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06187c83b94d03ae4ebb67a2873cf0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.768157 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bc7cfd006f4d4ca95541dab6873d32" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.771661 13058 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:53.772259 13058 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.772795 13245 tablet_bootstrap.cc:654] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:53.773403 13058 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.785714 13249 raft_consensus.cc:348] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.786007 13249 raft_consensus.cc:374] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.786067 13249 raft_consensus.cc:729] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.786296 13249 consensus_queue.cc:260] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.788765 14331 raft_consensus.cc:2726] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.788895 14421 ts_tablet_manager.cc:1888] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.789005 14421 log.cc:1196] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/1615f6167aae4dc096b9f563ab5e6215
I20240506 19:56:53.789362 14421 ts_tablet_manager.cc:1909] T 1615f6167aae4dc096b9f563ab5e6215 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:53.790045 14421 tablet_replica.cc:330] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:53.790233 14421 raft_consensus.cc:2229] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.790374 14421 raft_consensus.cc:2258] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.790975 14421 ts_tablet_manager.cc:1875] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:53.806382 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 237068us
W20240506 19:56:53.806978 12781 leader_election.cc:341] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b2a5a5e193f245de96cc82986e292f05
W20240506 19:56:53.807868 13057 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b2be78cbc8264f6e8d2e05137135161c
I20240506 19:56:53.807984 13057 leader_election.cc:302] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:53.808135 13057 raft_consensus.cc:2670] T b2be78cbc8264f6e8d2e05137135161c P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:53.812456 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116df9bd6ab04a30ba09861e765d7267" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.813438 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673dc7c83f8748a09c41d51b42214aaf" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.813977 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:53.814272 13122 raft_consensus.cc:3037] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:53.820402 12781 leader_election.cc:302] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:53.821066 14331 raft_consensus.cc:2726] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:53.819061 14436 raft_consensus.cc:482] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.826771 14442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:53.827302 14442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:53.828434 14199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 19:56:53.828486 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dafb661b2443f5afadc6816868bac8" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.832990 14421 ts_tablet_manager.cc:1888] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.830724 14436 raft_consensus.cc:504] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.833120 14421 log.cc:1196] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:53.833493 14421 ts_tablet_manager.cc:1909] T 971db6ac24d54efca95695ed1c90d4ed P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:53.838786 14436 leader_election.cc:288] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:53.840466 13244 tablet_bootstrap.cc:492] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:53.842597 13244 ts_tablet_manager.cc:1367] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.232s user 0.006s sys 0.004s
I20240506 19:56:53.845357 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b2a4720f6a46e4b0412a58c1e26bf7" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:53.835976 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bc7cfd006f4d4ca95541dab6873d32" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.837098 14439 tablet.cc:1722] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:53.840771 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4e97d10ae54e558920d116d8fa0ba5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:53.853149 13122 raft_consensus.cc:2454] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:53.866822 14421 tablet_replica.cc:330] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:53.867210 14421 raft_consensus.cc:2229] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.867508 14421 raft_consensus.cc:2258] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.868258 14421 ts_tablet_manager.cc:1875] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:53.873312 14401 raft_consensus.cc:482] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.873559 14401 raft_consensus.cc:504] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.874662 14401 leader_election.cc:288] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.882130 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d1f01c0edb4f6b8a621c1643ca1afb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.891227 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 1615f6167aae4dc096b9f563ab5e6215 (table default.Table_feb73e349178483fb087f43a269605b0 [id=b19563cb49834a49924bb9d06b60b6ed]) successfully deleted
W20240506 19:56:53.892848 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f1a792bf9b774b8f8cc5491605f504ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a792bf9b774b8f8cc5491605f504ac already in progress: creating tablet
W20240506 19:56:53.894057 12921 consensus_peers.cc:485] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:53.895175 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 971db6ac24d54efca95695ed1c90d4ed (table default.Table_cc8e3aa90fea4998b3e0a4a411add544 [id=86b683f4dec545118e95b30b194e1fa5]) successfully deleted
W20240506 19:56:53.895762 12921 consensus_peers.cc:485] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:53.897205 14383 raft_consensus.cc:482] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.897650 14383 raft_consensus.cc:504] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.898542 14383 leader_election.cc:288] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:53.899163 13249 ts_tablet_manager.cc:1398] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.297s user 0.012s sys 0.008s
I20240506 19:56:53.902894 13249 tablet_bootstrap.cc:492] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:53.901496 14460 raft_consensus.cc:482] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.901651 14383 raft_consensus.cc:482] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.904112 14383 raft_consensus.cc:504] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:53.902289 12954 tablet_service.cc:1467] Processing CreateTablet for tablet c09e90928a2442d9bb364e10bb3e405a (DEFAULT_TABLE table=default.Table_d59316fe0b304bd9b03da5f95408a1c6 [id=ac0916562c14439e91b1117812475f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.904657 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09e90928a2442d9bb364e10bb3e405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.904781 14383 leader_election.cc:288] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:53.899446 12921 consensus_peers.cc:485] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:53.903365 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 912fa5b484e74572861f58ce78b1572e (DEFAULT_TABLE table=default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.908473 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa5b484e74572861f58ce78b1572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.903892 12962 tablet_service.cc:1467] Processing CreateTablet for tablet b904072d89c6486fa3ca1557220f0b52 (DEFAULT_TABLE table=default.Table_7881939105214872a57c956985a14360 [id=c91b5a199c00499186aabe5137b4f59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.909857 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b904072d89c6486fa3ca1557220f0b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.904138 14460 raft_consensus.cc:504] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.902289 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 904f11fe9eac4fcea9832218a4bed98b (DEFAULT_TABLE table=default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.917677 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904f11fe9eac4fcea9832218a4bed98b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:53.918299 13058 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.918998 13245 tablet_bootstrap.cc:492] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:53.919128 13245 ts_tablet_manager.cc:1367] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.155s user 0.001s sys 0.026s
I20240506 19:56:53.919864 13245 raft_consensus.cc:348] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.920063 13245 raft_consensus.cc:374] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.920115 13245 raft_consensus.cc:729] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.920308 13245 consensus_queue.cc:260] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.920771 13245 ts_tablet_manager.cc:1398] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240506 19:56:53.921032 12921 consensus_peers.cc:485] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:53.921200 12921 consensus_peers.cc:485] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:53.924908 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df3ab9dff5148f9bb335a7209b29bbc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.925673 13245 tablet_bootstrap.cc:492] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:53.925882 13089 tablet_service.cc:1467] Processing CreateTablet for tablet e830e166e30d46edb77a2f5a30e4dac6 (DEFAULT_TABLE table=default.Table_6cc42687d6c545068a39c096f2932dcd [id=64268ea18e3b4fbb9037218bcca5bbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.926260 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e830e166e30d46edb77a2f5a30e4dac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:53.926975 13245 tablet_bootstrap.cc:654] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:53.936393 13058 consensus_peers.cc:485] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:53.936585 14401 raft_consensus.cc:482] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.936707 14401 raft_consensus.cc:504] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.937417 14401 leader_election.cc:288] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.938365 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0d827d732e4261a7a8edb549c1c8e3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:53.939021 13058 leader_election.cc:341] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.950835 13242 ts_tablet_manager.cc:1888] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.951022 13242 log.cc:1196] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ef03fcaa05d04ebd9f262f43ff7c02e4
I20240506 19:56:53.951447 13242 ts_tablet_manager.cc:1909] T ef03fcaa05d04ebd9f262f43ff7c02e4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:53.951814 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 802) took 3638 ms. Trace:
I20240506 19:56:53.951903 13242 rpcz_store.cc:276] 0506 19:56:50.313413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.313475 (+ 62us) service_pool.cc:225] Handling call
0506 19:56:53.724459 (+3410984us) ts_tablet_manager.cc:1120] Deleting tablet ef03fcaa05d04ebd9f262f43ff7c02e4
0506 19:56:53.724473 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.862533 (+138060us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.950693 (+ 88160us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:53.951793 (+ 1100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3410718}
I20240506 19:56:53.952622 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 820) took 3389 ms. Trace:
I20240506 19:56:53.952612 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ef03fcaa05d04ebd9f262f43ff7c02e4 (table default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]) successfully deleted
I20240506 19:56:53.952726 13242 rpcz_store.cc:276] 0506 19:56:50.563644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.563715 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:53.952284 (+3388569us) ts_tablet_manager.cc:1120] Deleting tablet 110a046c2db14758b8d28616ae63c18c
0506 19:56:53.952295 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.952603 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3388385}
I20240506 19:56:53.953096 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 821) took 3386 ms. Trace:
W20240506 19:56:53.953220 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 110a046c2db14758b8d28616ae63c18c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a046c2db14758b8d28616ae63c18c already in progress: creating tablet
I20240506 19:56:53.953179 13242 rpcz_store.cc:276] 0506 19:56:50.566381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.566436 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:53.952991 (+3386555us) ts_tablet_manager.cc:1120] Deleting tablet 269210fbc3d4446181b123f200aae472
0506 19:56:53.953000 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.953081 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3386382}
I20240506 19:56:53.953539 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 822) took 3386 ms. Trace:
I20240506 19:56:53.953617 13242 rpcz_store.cc:276] 0506 19:56:50.567512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.568523 (+ 1011us) service_pool.cc:225] Handling call
0506 19:56:53.953436 (+3384913us) ts_tablet_manager.cc:1120] Deleting tablet 0bc56299ed854d6c8ef201127692fd60
0506 19:56:53.953446 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.953525 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3384753}
W20240506 19:56:53.953688 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 269210fbc3d4446181b123f200aae472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 269210fbc3d4446181b123f200aae472 already in progress: creating tablet
I20240506 19:56:53.953964 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 824) took 3380 ms. Trace:
W20240506 19:56:53.954068 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0bc56299ed854d6c8ef201127692fd60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bc56299ed854d6c8ef201127692fd60 already in progress: creating tablet
I20240506 19:56:53.954037 13242 rpcz_store.cc:276] 0506 19:56:50.573235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.573288 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:53.953862 (+3380574us) ts_tablet_manager.cc:1120] Deleting tablet 1e64cb2f31b64b07848c2dfed18ab9f1
0506 19:56:53.953872 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.953948 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3380330}
I20240506 19:56:53.954391 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 823) took 3386 ms. Trace:
I20240506 19:56:53.954468 13242 rpcz_store.cc:276] 0506 19:56:50.568361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.597432 (+ 29071us) service_pool.cc:225] Handling call
0506 19:56:53.954293 (+3356861us) ts_tablet_manager.cc:1120] Deleting tablet 4c2eb5dac9d34fb09119868bf5a85803
0506 19:56:53.954302 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.954376 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3352619}
W20240506 19:56:53.954550 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1e64cb2f31b64b07848c2dfed18ab9f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e64cb2f31b64b07848c2dfed18ab9f1 already in progress: creating tablet
W20240506 19:56:53.955219 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4c2eb5dac9d34fb09119868bf5a85803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c2eb5dac9d34fb09119868bf5a85803 already in progress: creating tablet
I20240506 19:56:53.962467 14421 ts_tablet_manager.cc:1888] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.963281 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.963603 13246 tablet_replica.cc:330] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:53.963850 13246 raft_consensus.cc:2229] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:53.964023 13246 raft_consensus.cc:2258] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:53.964699 13246 ts_tablet_manager.cc:1875] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:53.965848 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f1a792bf9b774b8f8cc5491605f504ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a792bf9b774b8f8cc5491605f504ac already in progress: creating tablet
W20240506 19:56:53.970049 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 8bd1b51c240a4b1aad65839679f7bba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bd1b51c240a4b1aad65839679f7bba7 already in progress: creating tablet
I20240506 19:56:53.981627 13246 ts_tablet_manager.cc:1888] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:53.981848 13246 log.cc:1196] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/234e4f3439e84b5fbb2f7bab1dbd0822
I20240506 19:56:53.982218 13246 ts_tablet_manager.cc:1909] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:53.983263 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 17da01620b8b41008421b977455e0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.984787 13244 raft_consensus.cc:348] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.985021 13244 raft_consensus.cc:374] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:53.985081 13244 raft_consensus.cc:729] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:53.991428 13244 consensus_queue.cc:260] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.993583 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e8b8be953a4a5784f2d098fd6364e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.994256 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 392e4e4fa399443cb23b5641ff4c6c39 (DEFAULT_TABLE table=default.Table_c47cda70aad1490e8d9668e7240b0673 [id=8e65e9a5a18a4dbfb6a1b4ebfb76b166]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.994930 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:53.995833 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7c61244620487bbbac26dcdbf8a38c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:53.996809 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 766d77bb818e429fb9b745e99adea585 (DEFAULT_TABLE table=default.Table_2d668c000c934cf7906b877bc0938038 [id=c87e723310b64d9b8021bce63b144556]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.997241 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 04738def56324a368f8182543107e32e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.997622 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 62af97bfc4344c57b3b13ebd01f003be (DEFAULT_TABLE table=default.Table_6f73fec327d24cf7bfabb15d38171100 [id=f3ae423b282a4280a5aedc772a3725b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.997732 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 904f11fe9eac4fcea9832218a4bed98b (DEFAULT_TABLE table=default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.998250 14436 raft_consensus.cc:482] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:53.998404 14436 raft_consensus.cc:504] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:53.998476 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 912fa5b484e74572861f58ce78b1572e (DEFAULT_TABLE table=default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.998582 13089 tablet_service.cc:1467] Processing CreateTablet for tablet c09e90928a2442d9bb364e10bb3e405a (DEFAULT_TABLE table=default.Table_d59316fe0b304bd9b03da5f95408a1c6 [id=ac0916562c14439e91b1117812475f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.998739 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet d9443cdbaa3c40299b66d216673c4b1a with delete_type TABLET_DATA_DELETED (replaced by tablet ca23d6f0a9e0445cbd4d0e708c8de2d3) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.998777 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.999078 14436 leader_election.cc:288] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:53.999114 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet d7be9788df8e47d285bd932d078c6a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:53.999586 13096 tablet_service.cc:1467] Processing CreateTablet for tablet b904072d89c6486fa3ca1557220f0b52 (DEFAULT_TABLE table=default.Table_7881939105214872a57c956985a14360 [id=c91b5a199c00499186aabe5137b4f59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:53.999985 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.000111 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 452a83470585433e8def7174300d3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.000499 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:54.000502 12783 leader_election.cc:341] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 30ad527c3f2a4b53affa928e149db1df
I20240506 19:56:54.000814 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ebd7b994584b2bab6d27f493deab92) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.004115 13244 ts_tablet_manager.cc:1398] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.160s user 0.002s sys 0.000s
I20240506 19:56:54.010442 14421 log.cc:1196] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/8eb9296bb1104b4cbe279eecd3683796
I20240506 19:56:54.010848 14421 ts_tablet_manager.cc:1909] T 8eb9296bb1104b4cbe279eecd3683796 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:54.011998 14421 tablet_replica.cc:330] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:54.012202 14421 raft_consensus.cc:2229] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.012172 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 8eb9296bb1104b4cbe279eecd3683796 (table default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]) successfully deleted
I20240506 19:56:54.012337 14421 raft_consensus.cc:2258] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.013025 14421 ts_tablet_manager.cc:1875] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:53.988763 12922 consensus_peers.cc:485] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 796beb8e0a2c41d698bd4c05796263f1. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.014062 13249 tablet_bootstrap.cc:654] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:53.989728 12921 consensus_peers.cc:485] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.014412 12922 leader_election.cc:302] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:54.014487 12921 leader_election.cc:341] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:54.014665 12921 consensus_peers.cc:485] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.014683 14459 raft_consensus.cc:2781] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:54.014791 14459 raft_consensus.cc:686] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
W20240506 19:56:54.014883 12921 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.990972 14460 leader_election.cc:288] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:54.015390 13057 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.015513 13057 leader_election.cc:302] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.015822 14439 tablet.cc:1722] T d3ec577dd06640068575288c24e1105a P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240506 19:56:54.015921 13057 leader_election.cc:341] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.016021 13057 leader_election.cc:302] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.016378 14454 raft_consensus.cc:482] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:54.016374 13057 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.016459 13057 leader_election.cc:302] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.016486 14454 raft_consensus.cc:504] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:54.016757 13057 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:53.996475 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06187c83b94d03ae4ebb67a2873cf0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.016837 13057 leader_election.cc:302] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:54.017156 13057 leader_election.cc:341] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.017246 13057 leader_election.cc:302] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.017308 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b2a4720f6a46e4b0412a58c1e26bf7" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.017340 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0f74c7c4cd42d293fa244de3fd693b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.018551 12921 leader_election.cc:341] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.018999 14404 raft_consensus.cc:2726] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.019160 14404 raft_consensus.cc:2726] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.019244 14404 raft_consensus.cc:2726] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.019337 14404 raft_consensus.cc:2726] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.019439 14404 raft_consensus.cc:2726] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.020164 13246 tablet_replica.cc:330] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.020361 13246 raft_consensus.cc:2229] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:54.020612 13246 pending_rounds.cc:62] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Trying to abort 1 pending ops.
I20240506 19:56:54.020664 13246 pending_rounds.cc:68] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744091629129728 op_type: NO_OP noop_request { }
I20240506 19:56:54.020763 13246 raft_consensus.cc:2866] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:54.020846 13246 raft_consensus.cc:2258] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.021314 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0f74c7c4cd42d293fa244de3fd693b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:54.021535 13111 raft_consensus.cc:2454] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:53.954782 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 829) took 3330 ms. Trace:
I20240506 19:56:54.023561 13242 rpcz_store.cc:276] 0506 19:56:50.624118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.624175 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:53.954688 (+3330513us) ts_tablet_manager.cc:1120] Deleting tablet bd26e6b8115644cabe6532c7be1bbe95
0506 19:56:53.954697 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:53.954768 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3330312}
W20240506 19:56:54.024655 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bd26e6b8115644cabe6532c7be1bbe95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd26e6b8115644cabe6532c7be1bbe95 already in progress: creating tablet
I20240506 19:56:54.025185 13242 tablet_replica.cc:330] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:54.025528 13242 raft_consensus.cc:2229] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.026100 13242 raft_consensus.cc:2258] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.026562 14454 leader_election.cc:288] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.026739 13242 ts_tablet_manager.cc:1875] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.027447 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5217c257ff46e48c23563e496b6f08" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.028209 12921 leader_election.cc:341] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.031256 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d1f01c0edb4f6b8a621c1643ca1afb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:54.031965 13057 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.032127 13057 leader_election.cc:302] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.032609 14211 raft_consensus.cc:2726] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.033483 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0d827d732e4261a7a8edb549c1c8e3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.033823 12781 leader_election.cc:302] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.034528 14436 raft_consensus.cc:2726] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:54.034606 12781 leader_election.cc:341] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 30ad527c3f2a4b53affa928e149db1df
I20240506 19:56:54.034741 12781 leader_election.cc:302] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.035063 14436 raft_consensus.cc:2726] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.035799 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5217c257ff46e48c23563e496b6f08" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:54.036052 13120 raft_consensus.cc:2454] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:54.037710 12922 leader_election.cc:341] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:54.038110 12922 leader_election.cc:302] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.038707 14454 raft_consensus.cc:2726] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.026795 14459 consensus_queue.cc:237] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.040585 13244 tablet_bootstrap.cc:492] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.040768 13245 tablet_bootstrap.cc:492] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:54.040908 13245 ts_tablet_manager.cc:1367] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.021s
I20240506 19:56:54.042171 13245 raft_consensus.cc:348] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.046288 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392e4e4fa399443cb23b5641ff4c6c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.046437 12844 raft_consensus.cc:3037] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.049433 14436 raft_consensus.cc:482] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.049527 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766d77bb818e429fb9b745e99adea585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.049492 14454 raft_consensus.cc:482] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.049620 14454 raft_consensus.cc:504] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.049582 14436 raft_consensus.cc:504] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.050330 14436 leader_election.cc:288] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:54.012424 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 234e4f3439e84b5fbb2f7bab1dbd0822 (table default.Table_4cbe3de6b961432fbeacafa9ae5fe50c [id=f6e95966d94c4b5694b843e272ea941c]) successfully deleted
W20240506 19:56:54.051366 12783 leader_election.cc:341] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:54.051878 13244 tablet_bootstrap.cc:654] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.053496 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62af97bfc4344c57b3b13ebd01f003be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.057557 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904f11fe9eac4fcea9832218a4bed98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.061553 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912fa5b484e74572861f58ce78b1572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.069667 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b904072d89c6486fa3ca1557220f0b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.072274 14454 leader_election.cc:288] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:54.073554 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09e90928a2442d9bb364e10bb3e405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.074285 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973e600038742a6bec0bb2d8f3551fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.076745 12921 leader_election.cc:341] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.077456 14211 raft_consensus.cc:482] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.077586 14211 raft_consensus.cc:504] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.078176 14211 leader_election.cc:288] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.078912 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffb88e0a0c4125b04f4d8d328da3f5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.079280 14439 tablet.cc:1722] T fce272e5b5f6483c9d031dff1104401b P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240506 19:56:54.079546 13058 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.080582 14460 consensus_queue.cc:1035] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:54.081562 12922 consensus_peers.cc:485] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ceeae70e0d1144ea9e3e0f8727c15e6b. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:54.082252 12922 consensus_peers.cc:485] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c96f014293242be9640515697c0733b. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.082945 12922 leader_election.cc:302] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.083472 14460 raft_consensus.cc:2781] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:54.083561 14460 raft_consensus.cc:482] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:54.083619 14460 raft_consensus.cc:3037] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.086167 14460 raft_consensus.cc:504] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.086812 14460 leader_election.cc:288] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.087432 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0f74c7c4cd42d293fa244de3fd693b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:54.087603 14460 consensus_queue.cc:1035] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:54.088090 12921 leader_election.cc:341] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:54.088876 12922 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:56:54.089073 12922 leader_election.cc:302] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:54.070515 12508 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 55178.5us
I20240506 19:56:54.101600 14211 raft_consensus.cc:482] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.101814 14211 raft_consensus.cc:504] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.102658 14211 leader_election.cc:288] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.105361 13245 raft_consensus.cc:374] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.105361 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f9aaca444c839a88c07f58cb6ad6" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.105504 13245 raft_consensus.cc:729] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.105767 13245 consensus_queue.cc:260] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:54.106146 13058 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.106923 14460 raft_consensus.cc:2726] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.107087 13245 ts_tablet_manager.cc:1398] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
W20240506 19:56:54.108023 12922 consensus_peers.cc:485] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e90b93023e40b3bafbddf7bbcc10b2. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.108348 14460 consensus_queue.cc:1035] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:54.112468 12922 leader_election.cc:302] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.112798 14460 raft_consensus.cc:2781] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:54.112907 14460 raft_consensus.cc:482] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:54.113020 14460 raft_consensus.cc:3037] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.115725 14460 raft_consensus.cc:504] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.116796 14460 leader_election.cc:288] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.117208 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5217c257ff46e48c23563e496b6f08" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:54.117947 12921 leader_election.cc:341] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:54.124739 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 63096us
I20240506 19:56:54.126513 14474 tablet.cc:1722] T d3ec577dd06640068575288c24e1105a P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240506 19:56:54.129345 12921 consensus_peers.cc:485] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.131389 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f9aaca444c839a88c07f58cb6ad6" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.133214 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffb88e0a0c4125b04f4d8d328da3f5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.133771 13110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0f74c7c4cd42d293fa244de3fd693b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:54.133960 13110 raft_consensus.cc:3037] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.134085 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973e600038742a6bec0bb2d8f3551fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:54.134253 13120 raft_consensus.cc:2454] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:54.134935 13112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5217c257ff46e48c23563e496b6f08" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:54.135092 13112 raft_consensus.cc:3037] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.132603 14468 tablet.cc:1722] T fce272e5b5f6483c9d031dff1104401b P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240506 19:56:54.130307 13057 leader_election.cc:341] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:54.140707 12781 leader_election.cc:341] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:54.140833 12781 leader_election.cc:302] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.140952 12848 raft_consensus.cc:3037] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.141248 14436 raft_consensus.cc:2726] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.144294 14454 consensus_queue.cc:1035] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:54.145107 12922 leader_election.cc:302] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.147198 14383 raft_consensus.cc:2781] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:54.147289 14383 raft_consensus.cc:482] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:54.147359 14383 raft_consensus.cc:3037] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.148113 13249 tablet_bootstrap.cc:492] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:54.149917 14383 raft_consensus.cc:504] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.150827 14383 leader_election.cc:288] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:54.151579 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973e600038742a6bec0bb2d8f3551fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:54.151464 13246 ts_tablet_manager.cc:1875] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:54.152164 12921 leader_election.cc:341] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.156038 13115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973e600038742a6bec0bb2d8f3551fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:54.156241 13115 raft_consensus.cc:3037] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.165444 14468 tablet.cc:1722] T 4c6a2c98fd374b8cbc5a65069bd4cb5b P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240506 19:56:54.171026 12507 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet aee896bd7f14475a8bba1a5228d590da (table default.Table_c280dff5527d4a95ad135ceba07f8a7f [id=ae97ea5c98db4ea4baf292ee3eef4136], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet aee896bd7f14475a8bba1a5228d590da consensus state has no leader
I20240506 19:56:54.174654 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet a1160ab972124630a9be7abeefb4aa56 with delete_type TABLET_DATA_DELETED (replaced by tablet 44ce8749846847108edf5f554139cbd9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.175902 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet b2be78cbc8264f6e8d2e05137135161c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:52 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.176363 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 116df9bd6ab04a30ba09861e765d7267 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.179080 14478 tablet.cc:1722] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240506 19:56:54.181653 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61d8da431d09460ea4035343a15826ec has no consensus state
I20240506 19:56:54.185583 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet edaaead0cf2e4133b8bdec62fb4770a0
I20240506 19:56:54.186458 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet edaaead0cf2e4133b8bdec62fb4770a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:54.188056 13242 ts_tablet_manager.cc:1888] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.188192 13242 log.cc:1196] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a8b3ed0b56aa478abb44347afca8fcf6
I20240506 19:56:54.188177 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:54.188377 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_d12f952acabc497c9f8f644888a0a361 [id=1ef16dfbd249466fa6b0a7df43515b7b]). Expected version 3 got 0
I20240506 19:56:54.188472 13242 ts_tablet_manager.cc:1909] T a8b3ed0b56aa478abb44347afca8fcf6 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:54.188552 12522 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]). Expected version 2 got 0
I20240506 19:56:54.188776 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 837) took 3409 ms. Trace:
I20240506 19:56:54.188865 13242 rpcz_store.cc:276] 0506 19:56:50.778926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.778983 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:54.025107 (+3246124us) ts_tablet_manager.cc:1120] Deleting tablet a8b3ed0b56aa478abb44347afca8fcf6
0506 19:56:54.025124 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.184489 (+159365us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.187927 (+ 3438us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.188754 (+ 827us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3245912}
I20240506 19:56:54.189210 13242 tablet_replica.cc:330] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:54.189422 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a8b3ed0b56aa478abb44347afca8fcf6 (table default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]) successfully deleted
I20240506 19:56:54.189592 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f08ac26a65b94147960c37766cca1fe2" } new_table_name: "default.Table_efa08afa30fd48009749a76c90bd6725" modify_external_catalogs: true
I20240506 19:56:54.192754 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_24269238bd23445a93f9bf17765610af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.193441 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24269238bd23445a93f9bf17765610af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.195766 12514 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 986c31005f5c4ba3a2cf1fba8c39081c
I20240506 19:56:54.196306 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c2d2254e8afc4f8a80b105ceffab0b47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.196971 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2d2254e8afc4f8a80b105ceffab0b47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.199559 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1a1a7ae174ea4272afca27f6f28b6476" } new_table_name: "default.Table_b81df1dfbbdb4a029e76aace9fd4ac71" modify_external_catalogs: true
I20240506 19:56:54.202207 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d61d010cfd0a45fc9c7413c61e051f1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.203006 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d61d010cfd0a45fc9c7413c61e051f1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:54.205354 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06234a1df4264ddd9abbcaf88292795f consensus state has no leader
I20240506 19:56:54.205829 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f83a7797a045469bbc2a87de8b086a2a" } new_table_name: "default.Table_07bfadf060f9447e80912279571d2d43" modify_external_catalogs: true
I20240506 19:56:54.200496 14421 ts_tablet_manager.cc:1888] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.206678 14421 log.cc:1196] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:54.207098 14421 ts_tablet_manager.cc:1909] T 6b59ad067c274fddbba5f71e89480dec P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:54.208531 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ab095fce093c440cb77fc15d75f7af97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.209169 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab095fce093c440cb77fc15d75f7af97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:54.212291 12825 consensus_meta.cc:287] T 766d77bb818e429fb9b745e99adea585 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent flushing consensus metadata: real 0.504s user 0.008s sys 0.005s
W20240506 19:56:54.213055 12921 consensus_peers.cc:485] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.139103 13057 leader_election.cc:302] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:54.213624 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 83403.6us
W20240506 19:56:54.213977 13057 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.213968 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet edaaead0cf2e4133b8bdec62fb4770a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:54.214131 13057 leader_election.cc:302] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:54.214357 14470 log.cc:924] Time spent T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20240506 19:56:54.215166 14398 raft_consensus.cc:2726] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.216749 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_cae804c3fdd14f67bde6f49878f4d248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:54.217767 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet edaaead0cf2e4133b8bdec62fb4770a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:53 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.218138 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet 378759ca310f453fba45d48879e2dd1a with delete_type TABLET_DATA_DELETED (replaced by tablet f63e8e12fd3b453a96413f1fc769fb9f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:54.220281 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9ecd2e034e49e49772db7435da3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:54.149617 13249 ts_tablet_manager.cc:1367] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.247s user 0.012s sys 0.005s
I20240506 19:56:54.223176 13249 raft_consensus.cc:348] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.223393 13249 raft_consensus.cc:374] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.223448 13249 raft_consensus.cc:729] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:54.223887 13057 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.224015 13057 leader_election.cc:302] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:54.224583 13057 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.224674 13057 leader_election.cc:302] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.224862 14404 raft_consensus.cc:2726] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.224875 13242 raft_consensus.cc:2229] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.224932 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.225001 14404 raft_consensus.cc:2726] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.225050 13242 raft_consensus.cc:2258] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.222245 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fb3b18d9d707453393cfb058b284bfa6" } modify_external_catalogs: true
I20240506 19:56:54.224138 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 6b59ad067c274fddbba5f71e89480dec (table default.Table_37e9aef6398b47059569bab0a7265a94 [id=d8c76c1f826d49c4818b1e278d407b0e]) successfully deleted
I20240506 19:56:54.225179 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.225234 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fb3b18d9d707453393cfb058b284bfa6" } modify_external_catalogs: true
I20240506 19:56:54.225373 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.225543 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 08dafb661b2443f5afadc6816868bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:54.226169 14421 tablet_replica.cc:330] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:54.226708 14402 raft_consensus.cc:2726] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.227787 14421 raft_consensus.cc:2229] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.228225 14421 raft_consensus.cc:2258] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.228472 13242 ts_tablet_manager.cc:1875] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.229081 14421 ts_tablet_manager.cc:1875] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:54.236311 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cae804c3fdd14f67bde6f49878f4d248 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.237275 13244 tablet_bootstrap.cc:492] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.237485 13244 ts_tablet_manager.cc:1367] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.197s user 0.007s sys 0.001s
I20240506 19:56:54.238288 13244 raft_consensus.cc:348] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.238577 13244 raft_consensus.cc:374] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.238677 13244 raft_consensus.cc:729] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.238940 13244 consensus_queue.cc:260] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.239653 13244 ts_tablet_manager.cc:1398] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:54.240165 13245 tablet_bootstrap.cc:492] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:54.240814 13244 tablet_bootstrap.cc:492] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.242628 13244 tablet_bootstrap.cc:654] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.245628 13249 consensus_queue.cc:260] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.246238 13249 ts_tablet_manager.cc:1398] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240506 19:56:54.246793 14436 raft_consensus.cc:482] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.246937 14436 raft_consensus.cc:504] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.247614 14436 leader_election.cc:288] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:54.248113 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7380f259c68d4d7c89938a548533b354 has no consensus state
W20240506 19:56:54.261628 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ed6da25f3b94b26959e66e513088c16 consensus state has no leader
I20240506 19:56:54.262298 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_946dd68492dd4528a7925366e34068ba" } modify_external_catalogs: true
I20240506 19:56:54.262703 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_946dd68492dd4528a7925366e34068ba" } modify_external_catalogs: true
I20240506 19:56:54.263864 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2d668c000c934cf7906b877bc0938038" } modify_external_catalogs: true
I20240506 19:56:54.264106 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2d668c000c934cf7906b877bc0938038" } modify_external_catalogs: true
I20240506 19:56:54.265726 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 58c88f80f5b34672b7c3a4b604469bf1 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:54.267649 12783 leader_election.cc:341] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 034d6309de0740d0a5806b3847e334cb
I20240506 19:56:54.272186 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 65da8b15e0c6496bb86b47c49e348af0 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:54.272773 13110 raft_consensus.cc:2454] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:54.272536 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bc9e005eb0a4e518b182b219d5771ed consensus state has no leader
I20240506 19:56:54.273252 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 01da81cb9ee840ed9e6640af305eaecc from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:54.274647 13112 raft_consensus.cc:2454] T 8a5217c257ff46e48c23563e496b6f08 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:54.275183 12922 leader_election.cc:302] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.275458 13115 raft_consensus.cc:2454] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:54.275506 14383 raft_consensus.cc:2781] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:54.275655 14383 raft_consensus.cc:686] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:54.275900 14383 consensus_queue.cc:237] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.275972 12922 leader_election.cc:302] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.276571 14459 raft_consensus.cc:2781] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:54.276741 14459 raft_consensus.cc:686] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:54.277029 14459 consensus_queue.cc:237] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.278921 12922 leader_election.cc:302] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.279412 14454 raft_consensus.cc:2781] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:54.279537 14454 raft_consensus.cc:686] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:54.279815 14454 consensus_queue.cc:237] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.284461 13249 tablet_bootstrap.cc:492] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:54.286418 12781 leader_election.cc:341] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 034d6309de0740d0a5806b3847e334cb
I20240506 19:56:54.286543 13249 tablet_bootstrap.cc:654] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.286573 12781 leader_election.cc:302] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.286968 14436 raft_consensus.cc:2726] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.287433 14211 consensus_queue.cc:1035] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:54.291132 14436 raft_consensus.cc:482] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.291297 14436 raft_consensus.cc:504] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.292017 14436 leader_election.cc:288] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:54.292858 12781 leader_election.cc:341] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ac9ecd2e034e49e49772db7435da3c81
W20240506 19:56:54.295092 12783 leader_election.cc:341] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ac9ecd2e034e49e49772db7435da3c81
I20240506 19:56:54.295430 12783 leader_election.cc:302] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.295912 14436 raft_consensus.cc:2726] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.296250 14459 raft_consensus.cc:482] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.296411 14459 raft_consensus.cc:504] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.297263 13242 ts_tablet_manager.cc:1888] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.297379 13242 log.cc:1196] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/7d23f0a5ff6d430282256e9bddc186b9
I20240506 19:56:54.297658 14459 leader_election.cc:288] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.297729 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9443cdbaa3c40299b66d216673c4b1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.298226 12921 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.301476 14421 ts_tablet_manager.cc:1888] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.301847 14421 log.cc:1196] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/472c63d8995346d78633154a8f0d4c2b
I20240506 19:56:54.298269 13244 tablet_bootstrap.cc:492] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.301754 13242 ts_tablet_manager.cc:1909] T 7d23f0a5ff6d430282256e9bddc186b9 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:54.302486 14421 ts_tablet_manager.cc:1909] T 472c63d8995346d78633154a8f0d4c2b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:54.302717 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 838) took 3488 ms. Trace:
I20240506 19:56:54.302810 13242 rpcz_store.cc:276] 0506 19:56:50.813825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.820137 (+ 6312us) service_pool.cc:225] Handling call
0506 19:56:54.189166 (+3369029us) ts_tablet_manager.cc:1120] Deleting tablet 7d23f0a5ff6d430282256e9bddc186b9
0506 19:56:54.189175 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.232315 (+ 43140us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.297154 (+ 64839us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.302694 (+ 5540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3368799}
I20240506 19:56:54.303126 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 472c63d8995346d78633154a8f0d4c2b (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
I20240506 19:56:54.303370 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 844) took 3468 ms. Trace:
I20240506 19:56:54.303445 13242 rpcz_store.cc:276] 0506 19:56:50.834449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.847098 (+ 12649us) service_pool.cc:225] Handling call
0506 19:56:54.303242 (+3456144us) ts_tablet_manager.cc:1120] Deleting tablet 996eb97639464e0bb976f2b1ecbbb73b
0506 19:56:54.303252 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.303357 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3446626}
W20240506 19:56:54.305084 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet edaaead0cf2e4133b8bdec62fb4770a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edaaead0cf2e4133b8bdec62fb4770a0 already in progress: creating tablet
I20240506 19:56:54.305586 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 7d23f0a5ff6d430282256e9bddc186b9 (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
W20240506 19:56:54.305876 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 996eb97639464e0bb976f2b1ecbbb73b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996eb97639464e0bb976f2b1ecbbb73b already in progress: creating tablet
I20240506 19:56:54.306890 13246 ts_tablet_manager.cc:1888] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:54.307032 13246 log.cc:1196] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/17da01620b8b41008421b977455e0c9a
I20240506 19:56:54.307394 13246 ts_tablet_manager.cc:1909] T 17da01620b8b41008421b977455e0c9a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.307994 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 17da01620b8b41008421b977455e0c9a (table default.Table_e43a57ef696d4d749ff2f11fe74f999b [id=a445760bb4514251b6ff954ca946cd59]) successfully deleted
I20240506 19:56:54.308535 13244 ts_tablet_manager.cc:1367] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.002s
I20240506 19:56:54.309427 13244 raft_consensus.cc:348] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.309623 13244 raft_consensus.cc:374] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.309672 13244 raft_consensus.cc:729] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.310513 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 843) took 3476 ms. Trace:
I20240506 19:56:54.310603 13242 rpcz_store.cc:276] 0506 19:56:50.834315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.856740 (+ 22425us) service_pool.cc:225] Handling call
0506 19:56:54.310376 (+3453636us) ts_tablet_manager.cc:1120] Deleting tablet 5b15189e5cb54684a0f62308c2e5f131
0506 19:56:54.310387 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.310494 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3453402}
W20240506 19:56:54.311077 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5b15189e5cb54684a0f62308c2e5f131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b15189e5cb54684a0f62308c2e5f131 already in progress: creating tablet
I20240506 19:56:54.314710 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 840) took 3480 ms. Trace:
I20240506 19:56:54.314815 13242 rpcz_store.cc:276] 0506 19:56:50.834085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.857006 (+ 22921us) service_pool.cc:225] Handling call
0506 19:56:54.314553 (+3457547us) ts_tablet_manager.cc:1120] Deleting tablet b3e50bde3564493eb7267acb42cb8592
0506 19:56:54.314565 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.314688 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3457354}
W20240506 19:56:54.315344 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b3e50bde3564493eb7267acb42cb8592 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3e50bde3564493eb7267acb42cb8592 already in progress: creating tablet
I20240506 19:56:54.324434 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 766d77bb818e429fb9b745e99adea585
I20240506 19:56:54.325794 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 766d77bb818e429fb9b745e99adea585 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:54.325876 12964 tablet_service.cc:1514] Processing DeleteTablet for tablet 766d77bb818e429fb9b745e99adea585 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:54.329162 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 766d77bb818e429fb9b745e99adea585 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 766d77bb818e429fb9b745e99adea585 already in progress: creating tablet
I20240506 19:56:54.330299 13244 consensus_queue.cc:260] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.330965 13244 ts_tablet_manager.cc:1398] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20240506 19:56:54.331523 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 842) took 3497 ms. Trace:
I20240506 19:56:54.331616 13242 rpcz_store.cc:276] 0506 19:56:50.834228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.860941 (+ 26713us) service_pool.cc:225] Handling call
0506 19:56:54.331171 (+3470230us) ts_tablet_manager.cc:1120] Deleting tablet 9bb90bf767c94d73ace6411cfd31d65c
0506 19:56:54.331403 (+ 232us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.331506 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3448920}
W20240506 19:56:54.332069 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9bb90bf767c94d73ace6411cfd31d65c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bb90bf767c94d73ace6411cfd31d65c already in progress: creating tablet
I20240506 19:56:54.333508 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 841) took 3499 ms. Trace:
I20240506 19:56:54.333602 13242 rpcz_store.cc:276] 0506 19:56:50.834155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.861073 (+ 26918us) service_pool.cc:225] Handling call
0506 19:56:54.333395 (+3472322us) ts_tablet_manager.cc:1120] Deleting tablet 7f8989f02f25418ea9f824b3e5eeddd6
0506 19:56:54.333403 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.333489 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3460920}
W20240506 19:56:54.334043 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7f8989f02f25418ea9f824b3e5eeddd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f8989f02f25418ea9f824b3e5eeddd6 already in progress: creating tablet
I20240506 19:56:54.335515 13244 tablet_bootstrap.cc:492] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.336560 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 839) took 3502 ms. Trace:
I20240506 19:56:54.336655 13242 rpcz_store.cc:276] 0506 19:56:50.833988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.861191 (+ 27203us) service_pool.cc:225] Handling call
0506 19:56:54.336441 (+3475250us) ts_tablet_manager.cc:1120] Deleting tablet a03f574aec984b83ab596e92a6f323fc
0506 19:56:54.336450 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.336522 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3463868}
W20240506 19:56:54.337109 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a03f574aec984b83ab596e92a6f323fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a03f574aec984b83ab596e92a6f323fc already in progress: creating tablet
I20240506 19:56:54.345431 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 2919538141834fb6a83259e29288acdd from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:54.345888 13242 tablet_replica.cc:330] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:54.340610 13249 tablet_bootstrap.cc:492] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:54.346741 13242 raft_consensus.cc:2229] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.347115 13249 ts_tablet_manager.cc:1367] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20240506 19:56:54.350199 13249 raft_consensus.cc:348] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.350929 13242 raft_consensus.cc:2258] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.352222 13242 ts_tablet_manager.cc:1875] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.355640 14401 raft_consensus.cc:482] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.358592 13110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9443cdbaa3c40299b66d216673c4b1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:54.361546 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 67e8b8be953a4a5784f2d098fd6364e3 because the tablet was not found. No further retry: Not found: Tablet not found: 67e8b8be953a4a5784f2d098fd6364e3
I20240506 19:56:54.362378 13246 tablet_replica.cc:330] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.362848 13249 raft_consensus.cc:374] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.362963 13249 raft_consensus.cc:729] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.348765 13244 tablet_bootstrap.cc:654] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.373414 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 766d77bb818e429fb9b745e99adea585 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:54.373765 13058 consensus_peers.cc:485] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.374397 13245 tablet_bootstrap.cc:654] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.374411 13246 raft_consensus.cc:2229] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.374333 14401 raft_consensus.cc:504] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.374647 13246 raft_consensus.cc:2258] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.375257 13246 ts_tablet_manager.cc:1875] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.374346 13110 raft_consensus.cc:2454] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:54.375264 14401 leader_election.cc:288] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.375877 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.376397 13058 leader_election.cc:341] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.377257 13245 tablet_bootstrap.cc:492] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:54.377429 13245 ts_tablet_manager.cc:1367] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.000s
I20240506 19:56:54.377964 13244 tablet_bootstrap.cc:492] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.378140 13244 ts_tablet_manager.cc:1367] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20240506 19:56:54.378439 13245 raft_consensus.cc:348] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.378726 13245 raft_consensus.cc:374] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.378804 13245 raft_consensus.cc:729] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.379001 13244 raft_consensus.cc:348] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.379264 13244 raft_consensus.cc:374] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.379374 13244 raft_consensus.cc:729] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.379537 13245 consensus_queue.cc:260] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.379828 13244 consensus_queue.cc:260] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.380533 13244 ts_tablet_manager.cc:1398] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:54.381208 13244 tablet_bootstrap.cc:492] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.384209 13244 tablet_bootstrap.cc:654] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.384799 13245 ts_tablet_manager.cc:1398] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:54.385272 13245 tablet_bootstrap.cc:492] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:54.386946 13245 tablet_bootstrap.cc:654] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.391770 13249 consensus_queue.cc:260] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.392386 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.393112 13249 ts_tablet_manager.cc:1398] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20240506 19:56:54.393548 13249 tablet_bootstrap.cc:492] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:54.394232 12922 leader_election.cc:302] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.394956 14383 raft_consensus.cc:2781] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:54.395040 14383 raft_consensus.cc:482] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:54.395103 14383 raft_consensus.cc:3037] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.398298 13246 ts_tablet_manager.cc:1888] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.398458 13246 log.cc:1196] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/04738def56324a368f8182543107e32e
W20240506 19:56:54.398686 13057 leader_election.cc:341] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.398825 13246 ts_tablet_manager.cc:1909] T 04738def56324a368f8182543107e32e P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.398877 13057 leader_election.cc:302] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.399422 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 04738def56324a368f8182543107e32e (table default.Table_0c42cb0cd7e349fa86150517492387bb [id=52a0a57c5c19479e90684d073873755b]) successfully deleted
I20240506 19:56:54.399570 14401 raft_consensus.cc:2726] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.394703 14460 consensus_queue.cc:1035] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:54.406066 13249 tablet_bootstrap.cc:654] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.414577 14511 tablet.cc:1722] T fce272e5b5f6483c9d031dff1104401b P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:54.419096 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c4128d96524a4c20b2f9a7ceedccfe96" } modify_external_catalogs: true
I20240506 19:56:54.419368 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c4128d96524a4c20b2f9a7ceedccfe96" } modify_external_catalogs: true
I20240506 19:56:54.420207 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ef58202ae07041c18e495f8a92ec3eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.420791 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef58202ae07041c18e495f8a92ec3eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.425227 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_41de97bb590e4b7ebb026aad76af73b4" } new_table_name: "default.Table_4d9003ea123240c8806509c02e08ebe3" modify_external_catalogs: true
I20240506 19:56:54.427568 14459 raft_consensus.cc:482] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.427692 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_153effa513de4385b0ddfd1d561446ac" } modify_external_catalogs: true
I20240506 19:56:54.427747 14459 raft_consensus.cc:504] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.427850 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_153effa513de4385b0ddfd1d561446ac" } modify_external_catalogs: true
I20240506 19:56:54.433781 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_62d73cc2b6a847ebb8f40ad1ce877813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.434360 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d73cc2b6a847ebb8f40ad1ce877813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:54.440032 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc2c509a06554b9db6d5e0be97e159db consensus state has no leader
I20240506 19:56:54.440562 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_edb2261b3c4944cb8218f60533811bdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.441088 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edb2261b3c4944cb8218f60533811bdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.444747 13244 tablet_bootstrap.cc:492] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.444953 13244 ts_tablet_manager.cc:1367] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.005s
I20240506 19:56:54.447237 13244 raft_consensus.cc:348] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.447533 13244 raft_consensus.cc:374] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.447618 13244 raft_consensus.cc:729] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.447912 13244 consensus_queue.cc:260] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.448711 13244 ts_tablet_manager.cc:1398] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:54.449092 13244 tablet_bootstrap.cc:492] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.449437 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:54.451287 13244 tablet_bootstrap.cc:654] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.451364 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_10189ed452404f5d90cc93ca6b59e7e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.452231 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10189ed452404f5d90cc93ca6b59e7e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.454790 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fd07c6b21a494057a0f71bb60e892972" } new_table_name: "default.Table_1f4f4333e9c147919940536b669bffd3" modify_external_catalogs: true
I20240506 19:56:54.459736 13242 ts_tablet_manager.cc:1888] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.463840 13244 tablet_bootstrap.cc:492] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.468549 13244 ts_tablet_manager.cc:1367] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240506 19:56:54.468631 13242 log.cc:1196] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/104337875517445f862d82ece3a84889
I20240506 19:56:54.469513 13242 ts_tablet_manager.cc:1909] T 104337875517445f862d82ece3a84889 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:54.469692 13244 raft_consensus.cc:348] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.469970 13244 raft_consensus.cc:374] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.470057 13244 raft_consensus.cc:729] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.470337 13244 consensus_queue.cc:260] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.470516 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 845) took 3506 ms. Trace:
I20240506 19:56:54.470757 13242 rpcz_store.cc:276] 0506 19:56:50.963766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:50.963822 (+ 56us) service_pool.cc:225] Handling call
0506 19:56:54.341049 (+3377227us) ts_tablet_manager.cc:1120] Deleting tablet 104337875517445f862d82ece3a84889
0506 19:56:54.345803 (+ 4754us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.382464 (+ 36661us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.459598 (+ 77134us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.470441 (+ 10843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3377026}
I20240506 19:56:54.476744 14402 raft_consensus.cc:482] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.476928 14402 raft_consensus.cc:504] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.477720 14402 leader_election.cc:288] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.477802 13246 tablet_replica.cc:330] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.478006 13246 raft_consensus.cc:2229] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.478204 13246 raft_consensus.cc:2258] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.478830 13246 ts_tablet_manager.cc:1875] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.471009 13244 ts_tablet_manager.cc:1398] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:54.478940 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b9731a473449b19a9e95653a9d0830" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.483446 13244 tablet_bootstrap.cc:492] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.483470 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 104337875517445f862d82ece3a84889 (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]) successfully deleted
W20240506 19:56:54.483623 13058 leader_election.cc:341] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.485601 13242 tablet_replica.cc:330] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:54.485821 13242 raft_consensus.cc:2229] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.486011 13242 raft_consensus.cc:2258] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.486546 13242 ts_tablet_manager.cc:1875] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.491194 14436 raft_consensus.cc:482] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.491350 14436 raft_consensus.cc:504] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.491993 14436 leader_election.cc:288] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:54.493121 12783 leader_election.cc:341] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 7de7397db1e74b4488a3f9a70136bccb
I20240506 19:56:54.493324 13244 tablet_bootstrap.cc:654] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.503208 14459 leader_election.cc:288] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.503405 14459 consensus_queue.cc:1035] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:54.503690 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3435c11854649b7daf74f3d8c33d7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.504318 12921 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.509629 13245 tablet_bootstrap.cc:492] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:54.509819 13245 ts_tablet_manager.cc:1367] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.000s
I20240506 19:56:54.516496 13246 ts_tablet_manager.cc:1888] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.516652 13246 log.cc:1196] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:56:54.516682 14383 raft_consensus.cc:504] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.516989 13246 ts_tablet_manager.cc:1909] T d9443cdbaa3c40299b66d216673c4b1a P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.531242 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_fb5492528b144beeaec025a770bc00f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:54.531889 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb5492528b144beeaec025a770bc00f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:54.536902 14383 leader_election.cc:288] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.536798 13242 ts_tablet_manager.cc:1888] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.537114 13242 log.cc:1196] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/88dba275893f4696a5318ec21f0b2477
I20240506 19:56:54.537541 13242 ts_tablet_manager.cc:1909] T 88dba275893f4696a5318ec21f0b2477 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:54.538179 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9443cdbaa3c40299b66d216673c4b1a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:54.539163 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d9443cdbaa3c40299b66d216673c4b1a (table default.Table_456c37fe3dda48289f407577c6618ba9 [id=b164ee48c45e4826868242c03700533e]) successfully deleted
W20240506 19:56:54.539183 12921 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.542716 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 846) took 3462 ms. Trace:
I20240506 19:56:54.542852 13242 rpcz_store.cc:276] 0506 19:56:51.079991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.080065 (+ 74us) service_pool.cc:225] Handling call
0506 19:56:54.485529 (+3405464us) ts_tablet_manager.cc:1120] Deleting tablet 88dba275893f4696a5318ec21f0b2477
0506 19:56:54.485541 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.510290 (+ 24749us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.513255 (+ 2965us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.542662 (+ 29407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3405240}
I20240506 19:56:54.543329 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 88dba275893f4696a5318ec21f0b2477 (table default.Table_f4146ffd1ac744c4bdd3506d398dcdc0 [id=194197559e4e4469b5a3622465e90f2e]) successfully deleted
I20240506 19:56:54.543423 13245 raft_consensus.cc:348] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.543661 13245 raft_consensus.cc:374] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.543743 13245 raft_consensus.cc:729] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.543998 13245 consensus_queue.cc:260] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.544526 13245 ts_tablet_manager.cc:1398] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240506 19:56:54.548913 14511 tablet.cc:1722] T f75467ef60504763a67fc10e8639716c P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:54.551950 14468 tablet.cc:1722] T f75467ef60504763a67fc10e8639716c P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:54.554903 13246 tablet_replica.cc:330] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.555150 13246 raft_consensus.cc:2229] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.555332 13246 raft_consensus.cc:2258] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.555892 13246 ts_tablet_manager.cc:1875] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.558028 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 870) took 3366 ms. Trace:
I20240506 19:56:54.558151 13242 rpcz_store.cc:276] 0506 19:56:51.191983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.225689 (+ 33706us) service_pool.cc:225] Handling call
0506 19:56:54.557850 (+3332161us) ts_tablet_manager.cc:1120] Deleting tablet 2a3291123a1e42af80dbfb578c855618
0506 19:56:54.557860 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.557986 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3331387}
W20240506 19:56:54.558687 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2a3291123a1e42af80dbfb578c855618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a3291123a1e42af80dbfb578c855618 already in progress: creating tablet
I20240506 19:56:54.562713 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 871) took 3370 ms. Trace:
I20240506 19:56:54.563377 13242 rpcz_store.cc:276] 0506 19:56:51.192065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.226497 (+ 34432us) service_pool.cc:225] Handling call
0506 19:56:54.562545 (+3336048us) ts_tablet_manager.cc:1120] Deleting tablet 9c93117bdcf44c719c2e3588de538e96
0506 19:56:54.562556 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.562673 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3335907}
W20240506 19:56:54.564265 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9c93117bdcf44c719c2e3588de538e96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c93117bdcf44c719c2e3588de538e96 already in progress: creating tablet
W20240506 19:56:54.573973 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 71921.9us
I20240506 19:56:54.574549 13244 tablet_bootstrap.cc:492] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.574826 13244 ts_tablet_manager.cc:1367] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.003s
I20240506 19:56:54.576191 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 872) took 3384 ms. Trace:
I20240506 19:56:54.576766 14398 raft_consensus.cc:482] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.577006 14398 raft_consensus.cc:504] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:54.577463 12921 consensus_peers.cc:485] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.578087 14398 leader_election.cc:288] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.579524 13242 rpcz_store.cc:276] 0506 19:56:51.192135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.226652 (+ 34517us) service_pool.cc:225] Handling call
0506 19:56:54.576012 (+3349360us) ts_tablet_manager.cc:1120] Deleting tablet 5bbcc40911034f319f489931015ca257
0506 19:56:54.576023 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.576138 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3349235}
W20240506 19:56:54.580075 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5bbcc40911034f319f489931015ca257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bbcc40911034f319f489931015ca257 already in progress: creating tablet
I20240506 19:56:54.580535 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bece304e0ba048ecbb2d92aa71fdf707" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.580886 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 873) took 3388 ms. Trace:
I20240506 19:56:54.580996 13242 rpcz_store.cc:276] 0506 19:56:51.192237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.226784 (+ 34547us) service_pool.cc:225] Handling call
0506 19:56:54.580768 (+3353984us) ts_tablet_manager.cc:1120] Deleting tablet 2b3268094cbf481698cb373c1bf6dfba
0506 19:56:54.580776 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.580847 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3353867}
W20240506 19:56:54.581131 13058 leader_election.cc:341] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.584072 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_211f1d00bb6e407b90eac9c696f99e83" } new_table_name: "default.Table_23747e8c1b314e4eaa4566b557389261" modify_external_catalogs: true
I20240506 19:56:54.584594 13245 tablet_bootstrap.cc:492] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:54.581759 13244 raft_consensus.cc:348] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.585583 13244 raft_consensus.cc:374] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.585697 13244 raft_consensus.cc:729] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.586573 13245 tablet_bootstrap.cc:654] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.586617 13242 tablet_replica.cc:330] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:54.586819 13242 raft_consensus.cc:2229] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.587013 13242 raft_consensus.cc:2258] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:54.595762 12921 consensus_peers.cc:485] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.587569 13242 ts_tablet_manager.cc:1875] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.590622 13244 consensus_queue.cc:260] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.600095 13244 ts_tablet_manager.cc:1398] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20240506 19:56:54.600549 13244 tablet_bootstrap.cc:492] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.606557 13249 tablet_bootstrap.cc:492] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:54.608042 13244 tablet_bootstrap.cc:654] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.613776 13249 ts_tablet_manager.cc:1367] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.220s user 0.005s sys 0.012s
I20240506 19:56:54.614895 13249 raft_consensus.cc:348] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.615227 13249 raft_consensus.cc:374] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.615336 13249 raft_consensus.cc:729] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.615622 13249 consensus_queue.cc:260] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.616222 13249 ts_tablet_manager.cc:1398] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:54.616525 13249 tablet_bootstrap.cc:492] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:54.618172 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b9731a473449b19a9e95653a9d0830" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.618559 14518 raft_consensus.cc:482] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.618705 14518 raft_consensus.cc:504] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.619755 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6352d2c630204df3a24e299e167f92d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.620357 12921 leader_election.cc:341] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.620679 14518 leader_election.cc:288] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:54.628588 14436 raft_consensus.cc:482] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.628810 14436 raft_consensus.cc:504] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.629613 14436 leader_election.cc:288] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:54.630957 12783 leader_election.cc:341] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 378759ca310f453fba45d48879e2dd1a
I20240506 19:56:54.631475 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:54.632148 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bece304e0ba048ecbb2d92aa71fdf707" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:54.635653 12519 catalog_manager.cc:5484] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:54.636137 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:54.636376 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f8ce7f4ceb9d40009e9c75543b1fc545 (table default.Table_9068971de53c4d30bbe309692dd907f3 [id=7d2da7abfa3e48f79147c8a16820ebcb]). Expected version 1 got 0
I20240506 19:56:54.636579 12519 catalog_manager.cc:5484] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:54.637200 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:54.637605 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet fce272e5b5f6483c9d031dff1104401b (table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]). Expected version 3 got 0
I20240506 19:56:54.638031 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]). Expected version 2 got 0
I20240506 19:56:54.638278 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:54.638523 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:54.639307 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]). Expected version 1 got 0
I20240506 19:56:54.639569 12519 catalog_manager.cc:5484] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:54.640003 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]). Expected version 2 got 0
I20240506 19:56:54.640311 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:54.640568 12519 catalog_manager.cc:5484] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
W20240506 19:56:54.640772 12781 leader_election.cc:341] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 7de7397db1e74b4488a3f9a70136bccb
I20240506 19:56:54.640952 12781 leader_election.cc:302] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.641016 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet dcb357a7eb2147a79cc69d024bd35269 (table default.Table_b35bb26efd3c4ca08971e6c1185a6c0b [id=b6aeaf0e19d746dab4a37005dc753d65]). Expected version 1 got 0
I20240506 19:56:54.641315 12519 catalog_manager.cc:5484] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:54.641549 14436 raft_consensus.cc:2726] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.641788 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:54.642170 12519 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:54.642446 12516 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a] alter complete (version 3)
I20240506 19:56:54.601411 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3435c11854649b7daf74f3d8c33d7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:54.603296 14511 tablet.cc:1722] T d3ec577dd06640068575288c24e1105a P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:54.646767 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed98067bef848349d9dd93805674bbd (DEFAULT_TABLE table=default.Table_fb84f06ac8d34f659ca376315100395f [id=8b3fcd8705044aba983c11196817167f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.606102 14398 raft_consensus.cc:482] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:54.649844 12922 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:56:54.649981 12922 leader_election.cc:302] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.650430 14460 raft_consensus.cc:2726] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240506 19:56:54.652343 12922 consensus_peers.cc:485] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04738def56324a368f8182543107e32e. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:54.652499 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed98067bef848349d9dd93805674bbd (DEFAULT_TABLE table=default.Table_fb84f06ac8d34f659ca376315100395f [id=8b3fcd8705044aba983c11196817167f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.653393 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed98067bef848349d9dd93805674bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.659538 14533 tablet.cc:1722] T f75467ef60504763a67fc10e8639716c P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:54.662958 14532 tablet.cc:1722] T d380bf30599d4130bbb80b4f26837779 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:54.663690 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 22b9553636ec4943a045b6ae8017f7cc (DEFAULT_TABLE table=default.Table_eb945644f4344376aa5059f1a8223ce9 [id=18e06c57bec942d9b0c1339852f02dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.665096 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 22b9553636ec4943a045b6ae8017f7cc (DEFAULT_TABLE table=default.Table_eb945644f4344376aa5059f1a8223ce9 [id=18e06c57bec942d9b0c1339852f02dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.665493 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b9553636ec4943a045b6ae8017f7cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.666352 13249 tablet_bootstrap.cc:654] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.666668 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 8b28f732edd34b1286f012abf7b018fe (DEFAULT_TABLE table=default.Table_6b08752503f442de813189d8b7dea7b9 [id=256783d83a354331a1561cbb217a58ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.615994 13246 ts_tablet_manager.cc:1888] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:54.642119 13057 leader_election.cc:341] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.645233 13114 raft_consensus.cc:2454] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:54.665989 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 22b9553636ec4943a045b6ae8017f7cc (DEFAULT_TABLE table=default.Table_eb945644f4344376aa5059f1a8223ce9 [id=18e06c57bec942d9b0c1339852f02dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.667768 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b9553636ec4943a045b6ae8017f7cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:54.667948 12781 leader_election.cc:341] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 378759ca310f453fba45d48879e2dd1a
I20240506 19:56:54.668081 12781 leader_election.cc:302] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:54.668758 14436 raft_consensus.cc:2726] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.667152 13246 log.cc:1196] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:56:54.663082 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed98067bef848349d9dd93805674bbd (DEFAULT_TABLE table=default.Table_fb84f06ac8d34f659ca376315100395f [id=8b3fcd8705044aba983c11196817167f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.673846 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed98067bef848349d9dd93805674bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.674005 13246 ts_tablet_manager.cc:1909] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.674825 13246 tablet_replica.cc:330] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.675181 13246 raft_consensus.cc:2229] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.675508 13246 raft_consensus.cc:2258] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.677088 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 8b28f732edd34b1286f012abf7b018fe (DEFAULT_TABLE table=default.Table_6b08752503f442de813189d8b7dea7b9 [id=256783d83a354331a1561cbb217a58ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.667157 13057 leader_election.cc:302] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.677515 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b28f732edd34b1286f012abf7b018fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:54.677824 13057 leader_election.cc:341] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.677961 13057 leader_election.cc:302] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.678174 14404 raft_consensus.cc:2726] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.678476 14402 raft_consensus.cc:2726] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.681852 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 0aa8f9aaca444c839a88c07f58cb6ad6 (table default.Table_bef80a208c25432493da6ad772c8bf32 [id=b759e15bf3d341939e506cb620dd5448]) successfully deleted
I20240506 19:56:54.681453 13246 ts_tablet_manager.cc:1875] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.682291 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 8b28f732edd34b1286f012abf7b018fe (DEFAULT_TABLE table=default.Table_6b08752503f442de813189d8b7dea7b9 [id=256783d83a354331a1561cbb217a58ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.682757 14474 tablet.cc:1722] T d380bf30599d4130bbb80b4f26837779 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:54.683203 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b28f732edd34b1286f012abf7b018fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.683755 12922 leader_election.cc:302] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.684015 14383 raft_consensus.cc:2781] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:54.684072 14383 raft_consensus.cc:482] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:54.684128 14383 raft_consensus.cc:3037] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.686012 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 61d8da431d09460ea4035343a15826ec (DEFAULT_TABLE table=default.Table_d2889007e246431bb796c67ca72db0aa [id=63af27bdf8c84d478759499c8437682c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.686367 14383 raft_consensus.cc:504] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.687462 14383 leader_election.cc:288] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.687723 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3435c11854649b7daf74f3d8c33d7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:54.688228 12921 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.689173 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3435c11854649b7daf74f3d8c33d7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:54.689409 13111 raft_consensus.cc:3037] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.690032 14511 tablet.cc:1722] T f8ce7f4ceb9d40009e9c75543b1fc545 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:54.690819 13244 tablet_bootstrap.cc:492] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.690940 13244 ts_tablet_manager.cc:1367] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.004s
I20240506 19:56:54.692018 13111 raft_consensus.cc:2454] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:54.691948 13244 raft_consensus.cc:348] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.642481 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6352d2c630204df3a24e299e167f92d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:54.692181 13244 raft_consensus.cc:374] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.692279 13244 raft_consensus.cc:729] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.692518 13244 consensus_queue.cc:260] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.693187 13244 ts_tablet_manager.cc:1398] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:54.693364 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 61d8da431d09460ea4035343a15826ec (DEFAULT_TABLE table=default.Table_d2889007e246431bb796c67ca72db0aa [id=63af27bdf8c84d478759499c8437682c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.693536 13244 tablet_bootstrap.cc:492] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.693796 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d8da431d09460ea4035343a15826ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.694154 12922 leader_election.cc:302] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.695026 14383 raft_consensus.cc:2781] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:54.695142 14383 raft_consensus.cc:686] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:54.695312 13244 tablet_bootstrap.cc:654] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.695410 14383 consensus_queue.cc:237] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.696691 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7aceb691114827821194eb4b6760fc (DEFAULT_TABLE table=default.Table_7297af847f20409aae62497c2b197328 [id=edf335649ed04b0a9027ba71825918ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.700918 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 61d8da431d09460ea4035343a15826ec (DEFAULT_TABLE table=default.Table_d2889007e246431bb796c67ca72db0aa [id=63af27bdf8c84d478759499c8437682c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.701649 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d8da431d09460ea4035343a15826ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.701176 13116 raft_consensus.cc:2454] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:54.649322 14398 raft_consensus.cc:504] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.705500 14398 leader_election.cc:288] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.706434 12922 leader_election.cc:302] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.706449 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:54.707033 13057 leader_election.cc:341] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.707355 14080 raft_consensus.cc:2781] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:54.707466 14080 raft_consensus.cc:482] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:54.707572 14080 raft_consensus.cc:3037] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.710042 14080 raft_consensus.cc:504] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.701925 12964 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7aceb691114827821194eb4b6760fc (DEFAULT_TABLE table=default.Table_7297af847f20409aae62497c2b197328 [id=edf335649ed04b0a9027ba71825918ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.711222 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.711952 13058 leader_election.cc:341] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.712095 13058 leader_election.cc:302] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:54.712590 14398 raft_consensus.cc:2726] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:54.712941 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6352d2c630204df3a24e299e167f92d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:54.713496 12921 leader_election.cc:341] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.713199 14539 tablet.cc:1722] T d380bf30599d4130bbb80b4f26837779 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:54.722107 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7aceb691114827821194eb4b6760fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.726174 14538 tablet.cc:1722] T f8ce7f4ceb9d40009e9c75543b1fc545 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:54.729387 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7aceb691114827821194eb4b6760fc (DEFAULT_TABLE table=default.Table_7297af847f20409aae62497c2b197328 [id=edf335649ed04b0a9027ba71825918ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.730907 12827 tablet_service.cc:1467] Processing CreateTablet for tablet c37c9f16db9b43bb8d2b9d924b596a1e (DEFAULT_TABLE table=default.Table_acd9eb17729b45d69cd7e0b55fb9e63a [id=959ae33860b1462eb07511168b279534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.731905 14511 tablet.cc:1722] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:54.732551 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7aceb691114827821194eb4b6760fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.732704 14080 leader_election.cc:288] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:54.705247 14532 tablet.cc:1722] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:54.737397 13244 tablet_bootstrap.cc:492] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.737628 13244 ts_tablet_manager.cc:1367] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
W20240506 19:56:54.737850 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f4e97d10ae54e558920d116d8fa0ba5 consensus state has no leader
I20240506 19:56:54.738272 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b81df1dfbbdb4a029e76aace9fd4ac71" } new_table_name: "default.Table_af258c3a0d0a466ea0d34bbfae91eb67" modify_external_catalogs: true
I20240506 19:56:54.738592 13244 raft_consensus.cc:348] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.739065 13244 raft_consensus.cc:374] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.739125 13244 raft_consensus.cc:729] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.739351 13244 consensus_queue.cc:260] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.739784 13244 ts_tablet_manager.cc:1398] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:54.739990 13249 tablet_bootstrap.cc:492] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:54.740041 13244 tablet_bootstrap.cc:492] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:54.740193 13249 ts_tablet_manager.cc:1367] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.002s
I20240506 19:56:54.740955 13249 raft_consensus.cc:348] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.741559 13249 raft_consensus.cc:374] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.741607 13249 raft_consensus.cc:729] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.741848 13249 consensus_queue.cc:260] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.742033 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed98067bef848349d9dd93805674bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.742276 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b28f732edd34b1286f012abf7b018fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.742298 13249 ts_tablet_manager.cc:1398] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:54.742530 13249 tablet_bootstrap.cc:492] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:54.744220 13249 tablet_bootstrap.cc:654] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.745335 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b9553636ec4943a045b6ae8017f7cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.747530 13244 tablet_bootstrap.cc:654] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.749511 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d8da431d09460ea4035343a15826ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.753480 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37c9f16db9b43bb8d2b9d924b596a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.755715 13244 tablet_bootstrap.cc:492] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:54.755844 13244 ts_tablet_manager.cc:1367] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240506 19:56:54.756322 13249 tablet_bootstrap.cc:492] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:54.756450 13249 ts_tablet_manager.cc:1367] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20240506 19:56:54.756630 13244 raft_consensus.cc:348] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.756798 13244 raft_consensus.cc:374] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.756850 13244 raft_consensus.cc:729] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.757086 13244 consensus_queue.cc:260] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.757762 13244 ts_tablet_manager.cc:1398] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:54.757750 13249 raft_consensus.cc:348] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.757946 13249 raft_consensus.cc:374] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.758011 13249 raft_consensus.cc:729] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.758065 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7aceb691114827821194eb4b6760fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.758236 13249 consensus_queue.cc:260] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.758971 13249 ts_tablet_manager.cc:1398] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:54.759264 13249 tablet_bootstrap.cc:492] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:54.760702 13249 tablet_bootstrap.cc:654] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.741251 13245 tablet_bootstrap.cc:492] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:54.763325 13245 ts_tablet_manager.cc:1367] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.179s user 0.004s sys 0.000s
I20240506 19:56:54.768528 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6352d2c630204df3a24e299e167f92d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:54.774641 13114 raft_consensus.cc:3037] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:54.775252 14383 raft_consensus.cc:482] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.775408 14383 raft_consensus.cc:504] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.776017 14383 leader_election.cc:288] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.776960 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234e4f3439e84b5fbb2f7bab1dbd0822" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.777588 13114 raft_consensus.cc:2454] T 6352d2c630204df3a24e299e167f92d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:54.777801 12921 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.778896 13245 raft_consensus.cc:348] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.779183 13245 raft_consensus.cc:374] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.779289 13245 raft_consensus.cc:729] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.779563 13245 consensus_queue.cc:260] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.780596 12954 tablet_service.cc:1467] Processing CreateTablet for tablet c37c9f16db9b43bb8d2b9d924b596a1e (DEFAULT_TABLE table=default.Table_acd9eb17729b45d69cd7e0b55fb9e63a [id=959ae33860b1462eb07511168b279534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.781066 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37c9f16db9b43bb8d2b9d924b596a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.781116 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 7f41319b312d48c6bd921641086c7b5a (DEFAULT_TABLE table=default.Table_31d441acbe354b93b7056bee4a33a55a [id=746269304afa4d889e5a0eedb55e192b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.782100 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 041612b3636a49db9e303877964a31ff (DEFAULT_TABLE table=default.Table_dbd73da3dbb94cc59750f8b4d27a46ee [id=a3a24a54b336458d8e6c2014ab3b38f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.782898 12820 tablet_service.cc:1467] Processing CreateTablet for tablet a41aa8aa4c9a4163b1eae92f11a74b46 (DEFAULT_TABLE table=default.Table_b35d5d1b38604d24bf72370ec6d5998c [id=b992a884276341348ee0926305500131]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.783589 13089 tablet_service.cc:1467] Processing CreateTablet for tablet a41aa8aa4c9a4163b1eae92f11a74b46 (DEFAULT_TABLE table=default.Table_b35d5d1b38604d24bf72370ec6d5998c [id=b992a884276341348ee0926305500131]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.784029 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41aa8aa4c9a4163b1eae92f11a74b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.789923 12819 tablet_service.cc:1467] Processing CreateTablet for tablet c2441d75b9bc48409aa0022d6a30d8c8 (DEFAULT_TABLE table=default.Table_44b86b8ac10f4b758b4356ccf982efb0 [id=f300837e10554f88bddd6e6e6bce630b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.790694 13097 tablet_service.cc:1467] Processing CreateTablet for tablet c2441d75b9bc48409aa0022d6a30d8c8 (DEFAULT_TABLE table=default.Table_44b86b8ac10f4b758b4356ccf982efb0 [id=f300837e10554f88bddd6e6e6bce630b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.791119 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2441d75b9bc48409aa0022d6a30d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.791036 12960 tablet_service.cc:1467] Processing CreateTablet for tablet c2441d75b9bc48409aa0022d6a30d8c8 (DEFAULT_TABLE table=default.Table_44b86b8ac10f4b758b4356ccf982efb0 [id=f300837e10554f88bddd6e6e6bce630b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.791280 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 041612b3636a49db9e303877964a31ff (DEFAULT_TABLE table=default.Table_dbd73da3dbb94cc59750f8b4d27a46ee [id=a3a24a54b336458d8e6c2014ab3b38f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.791400 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2441d75b9bc48409aa0022d6a30d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.791572 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041612b3636a49db9e303877964a31ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.791033 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 7f41319b312d48c6bd921641086c7b5a (DEFAULT_TABLE table=default.Table_31d441acbe354b93b7056bee4a33a55a [id=746269304afa4d889e5a0eedb55e192b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.792058 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41319b312d48c6bd921641086c7b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.797341 12966 tablet_service.cc:1467] Processing CreateTablet for tablet a41aa8aa4c9a4163b1eae92f11a74b46 (DEFAULT_TABLE table=default.Table_b35d5d1b38604d24bf72370ec6d5998c [id=b992a884276341348ee0926305500131]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.797686 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41aa8aa4c9a4163b1eae92f11a74b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.797991 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7f41319b312d48c6bd921641086c7b5a (DEFAULT_TABLE table=default.Table_31d441acbe354b93b7056bee4a33a55a [id=746269304afa4d889e5a0eedb55e192b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.798446 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41319b312d48c6bd921641086c7b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.800797 13246 ts_tablet_manager.cc:1888] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.800918 13246 log.cc:1196] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d7be9788df8e47d285bd932d078c6a8b
I20240506 19:56:54.801501 13246 ts_tablet_manager.cc:1909] T d7be9788df8e47d285bd932d078c6a8b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.801929 13246 tablet_replica.cc:330] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.802126 13246 raft_consensus.cc:2229] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.802359 13246 raft_consensus.cc:2258] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.802804 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d7be9788df8e47d285bd932d078c6a8b (table default.Table_75c7fd329faa49bf99581532e6fd75d2 [id=8baacc24dda44dd5ad46aef596e96276]) successfully deleted
I20240506 19:56:54.803125 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 041612b3636a49db9e303877964a31ff (DEFAULT_TABLE table=default.Table_dbd73da3dbb94cc59750f8b4d27a46ee [id=a3a24a54b336458d8e6c2014ab3b38f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.803810 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041612b3636a49db9e303877964a31ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.805667 14436 raft_consensus.cc:482] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.805817 14436 raft_consensus.cc:504] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.806468 14436 leader_election.cc:288] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.807052 13085 tablet_service.cc:1467] Processing CreateTablet for tablet c37c9f16db9b43bb8d2b9d924b596a1e (DEFAULT_TABLE table=default.Table_acd9eb17729b45d69cd7e0b55fb9e63a [id=959ae33860b1462eb07511168b279534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.807498 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37c9f16db9b43bb8d2b9d924b596a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.809414 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 4a369e21e62444208be40d8c552b1626 (DEFAULT_TABLE table=default.Table_f9f1fa97c6434970b181f7536994782e [id=be8981747042401ebf0f6f583b71d220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.810132 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:54.810628 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 4a369e21e62444208be40d8c552b1626 (DEFAULT_TABLE table=default.Table_f9f1fa97c6434970b181f7536994782e [id=be8981747042401ebf0f6f583b71d220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.811262 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 4a369e21e62444208be40d8c552b1626 (DEFAULT_TABLE table=default.Table_f9f1fa97c6434970b181f7536994782e [id=be8981747042401ebf0f6f583b71d220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.811641 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a369e21e62444208be40d8c552b1626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.817768 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a369e21e62444208be40d8c552b1626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.818562 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 5024b8379db74d278f91894fad290691 (DEFAULT_TABLE table=default.Table_6fb4b0e1d5c94d88b7f9792005908344 [id=0206d0193f404359bf21bc2d8662c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.819911 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 5024b8379db74d278f91894fad290691 (DEFAULT_TABLE table=default.Table_6fb4b0e1d5c94d88b7f9792005908344 [id=0206d0193f404359bf21bc2d8662c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.820050 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 5024b8379db74d278f91894fad290691 (DEFAULT_TABLE table=default.Table_6fb4b0e1d5c94d88b7f9792005908344 [id=0206d0193f404359bf21bc2d8662c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.820529 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5024b8379db74d278f91894fad290691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.820927 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 79f643b5e89e470aa35f258109bbc22f (DEFAULT_TABLE table=default.Table_742be09ecdef4a4c9d11965a17109e81 [id=82062552076642299d2d7e511dd6bebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.821704 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 79f643b5e89e470aa35f258109bbc22f (DEFAULT_TABLE table=default.Table_742be09ecdef4a4c9d11965a17109e81 [id=82062552076642299d2d7e511dd6bebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.821913 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 79f643b5e89e470aa35f258109bbc22f (DEFAULT_TABLE table=default.Table_742be09ecdef4a4c9d11965a17109e81 [id=82062552076642299d2d7e511dd6bebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.822208 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9fa00ac9f645359b51384fba7179bf (DEFAULT_TABLE table=default.Table_540b5b87de2e447a8ce8a0b8f758cab7 [id=e094cbf306144ec3a64a14f9feb83085]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.822305 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f643b5e89e470aa35f258109bbc22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.823302 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9fa00ac9f645359b51384fba7179bf (DEFAULT_TABLE table=default.Table_540b5b87de2e447a8ce8a0b8f758cab7 [id=e094cbf306144ec3a64a14f9feb83085]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.823652 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9fa00ac9f645359b51384fba7179bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.823897 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe159e09e324e4bac409d2d6269292f (DEFAULT_TABLE table=default.Table_62d8dd9a1336466d8d008ec2ca4ff8a0 [id=788f6eb22d6e41cfbe0fb6144bda15f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.824327 13246 ts_tablet_manager.cc:1875] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.825657 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9fa00ac9f645359b51384fba7179bf (DEFAULT_TABLE table=default.Table_540b5b87de2e447a8ce8a0b8f758cab7 [id=e094cbf306144ec3a64a14f9feb83085]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.826812 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe159e09e324e4bac409d2d6269292f (DEFAULT_TABLE table=default.Table_62d8dd9a1336466d8d008ec2ca4ff8a0 [id=788f6eb22d6e41cfbe0fb6144bda15f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.827430 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac (DEFAULT_TABLE table=default.Table_cf12763862094c8197c09b72b43f4616 [id=cf9c889d67414705b6bda662c1bc3e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.828969 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac (DEFAULT_TABLE table=default.Table_cf12763862094c8197c09b72b43f4616 [id=cf9c889d67414705b6bda662c1bc3e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.830042 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 7380f259c68d4d7c89938a548533b354 (DEFAULT_TABLE table=default.Table_efa08afa30fd48009749a76c90bd6725 [id=0df30d7b42ff4869ad37a24a82300280]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.830056 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac (DEFAULT_TABLE table=default.Table_cf12763862094c8197c09b72b43f4616 [id=cf9c889d67414705b6bda662c1bc3e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.830099 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe159e09e324e4bac409d2d6269292f (DEFAULT_TABLE table=default.Table_62d8dd9a1336466d8d008ec2ca4ff8a0 [id=788f6eb22d6e41cfbe0fb6144bda15f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.830433 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.831140 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe159e09e324e4bac409d2d6269292f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.834394 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.835680 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 7380f259c68d4d7c89938a548533b354 (DEFAULT_TABLE table=default.Table_efa08afa30fd48009749a76c90bd6725 [id=0df30d7b42ff4869ad37a24a82300280]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.836112 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7380f259c68d4d7c89938a548533b354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.836103 12811 tablet_service.cc:1467] Processing CreateTablet for tablet c98d8aed6bc8460b9be6b8bfd00373d9 (DEFAULT_TABLE table=default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.837570 13104 tablet_service.cc:1467] Processing CreateTablet for tablet c98d8aed6bc8460b9be6b8bfd00373d9 (DEFAULT_TABLE table=default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.837913 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98d8aed6bc8460b9be6b8bfd00373d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.837975 12823 tablet_service.cc:1467] Processing CreateTablet for tablet d4dfe8f732074180a159027013cec014 (DEFAULT_TABLE table=default.Table_bbde4cb2e81b4723a42d8d0c2006c1a0 [id=3ea3309b8a554d64ab44ae10da633751]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.840718 13249 tablet_bootstrap.cc:492] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:54.840868 13249 ts_tablet_manager.cc:1367] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.082s user 0.020s sys 0.015s
I20240506 19:56:54.840945 13092 tablet_service.cc:1467] Processing CreateTablet for tablet d4dfe8f732074180a159027013cec014 (DEFAULT_TABLE table=default.Table_bbde4cb2e81b4723a42d8d0c2006c1a0 [id=3ea3309b8a554d64ab44ae10da633751]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.841341 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dfe8f732074180a159027013cec014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.841522 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5024b8379db74d278f91894fad290691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.841477 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 060cfeedba024cebb19ccccb49d4d582 (DEFAULT_TABLE table=default.Table_f9d1e5113608496eb4aad3218eb44a65 [id=f7188aea4c354dd4920e4fd94f6b5323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.841581 13249 raft_consensus.cc:348] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.841770 13249 raft_consensus.cc:374] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:54.841828 13249 raft_consensus.cc:729] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:54.842039 13249 consensus_queue.cc:260] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.842669 13249 ts_tablet_manager.cc:1398] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:54.842916 13249 tablet_bootstrap.cc:492] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:54.844370 13249 tablet_bootstrap.cc:654] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:54.846019 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f643b5e89e470aa35f258109bbc22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.849334 13088 tablet_service.cc:1467] Processing CreateTablet for tablet a48e1bae892a41caa69609dd89700939 (DEFAULT_TABLE table=default.Table_c1b76d968cf047ab833ec4ec4af83697 [id=2c499becf4f44f84bc6d33b9d92d8791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.849720 14518 raft_consensus.cc:482] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.849887 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9fa00ac9f645359b51384fba7179bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.849838 14518 raft_consensus.cc:504] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.850368 13246 ts_tablet_manager.cc:1888] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.850447 14518 leader_election.cc:288] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:54.850520 13246 log.cc:1196] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/997ecba7fbae41178a25bf7af4d152e1
I20240506 19:56:54.850478 13103 tablet_service.cc:1467] Processing CreateTablet for tablet a02b198026be469282f8dc0e82181973 (DEFAULT_TABLE table=default.Table_68c8729d5e134c10be8d88e246c4e00d [id=4537e3ced96b436a9e4d134427351495]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.850878 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a02b198026be469282f8dc0e82181973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.850908 13246 ts_tablet_manager.cc:1909] T 997ecba7fbae41178a25bf7af4d152e1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.851393 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7be9788df8e47d285bd932d078c6a8b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:54.851984 12921 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.852567 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 934ea08878b84c6c8e5ef7ca0d3fa65c (DEFAULT_TABLE table=default.Table_33e9d54058954ad9806125313b789621 [id=6cab7f5bc39a44e187c9098a05c2e4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.853588 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe159e09e324e4bac409d2d6269292f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.853729 13091 tablet_service.cc:1467] Processing CreateTablet for tablet e726aa52406d49d3b5c78f558755ac6e (DEFAULT_TABLE table=default.Table_6c36d4fe94e7454280ad13d12a5ed6db [id=e354c012edc64f49bdc0ab0dde85d05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.854521 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 182a3de861984b7d978f15a17275b6b0 (DEFAULT_TABLE table=default.Table_153effa513de4385b0ddfd1d561446ac [id=afed6df2d7b14d34b8274cc71e15c12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.858731 12964 tablet_service.cc:1467] Processing CreateTablet for tablet c98d8aed6bc8460b9be6b8bfd00373d9 (DEFAULT_TABLE table=default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.858805 12960 tablet_service.cc:1467] Processing CreateTablet for tablet d4dfe8f732074180a159027013cec014 (DEFAULT_TABLE table=default.Table_bbde4cb2e81b4723a42d8d0c2006c1a0 [id=3ea3309b8a554d64ab44ae10da633751]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.859084 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98d8aed6bc8460b9be6b8bfd00373d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.859257 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 060cfeedba024cebb19ccccb49d4d582 (DEFAULT_TABLE table=default.Table_f9d1e5113608496eb4aad3218eb44a65 [id=f7188aea4c354dd4920e4fd94f6b5323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.859520 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060cfeedba024cebb19ccccb49d4d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.861542 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 7380f259c68d4d7c89938a548533b354 (DEFAULT_TABLE table=default.Table_efa08afa30fd48009749a76c90bd6725 [id=0df30d7b42ff4869ad37a24a82300280]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.861851 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7380f259c68d4d7c89938a548533b354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.864820 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dfe8f732074180a159027013cec014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.861543 12967 tablet_service.cc:1467] Processing CreateTablet for tablet a48e1bae892a41caa69609dd89700939 (DEFAULT_TABLE table=default.Table_c1b76d968cf047ab833ec4ec4af83697 [id=2c499becf4f44f84bc6d33b9d92d8791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.865628 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48e1bae892a41caa69609dd89700939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.869622 13242 ts_tablet_manager.cc:1888] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.871680 12964 tablet_service.cc:1467] Processing CreateTablet for tablet e726aa52406d49d3b5c78f558755ac6e (DEFAULT_TABLE table=default.Table_6c36d4fe94e7454280ad13d12a5ed6db [id=e354c012edc64f49bdc0ab0dde85d05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.871894 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 4a49d621e94c4eadb63be52e58c3fab5 (DEFAULT_TABLE table=default.Table_4dcb29fa5425483296ab644d95581803 [id=c8a04b6ee94a46c4995bbd1182157875]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.872082 12950 tablet_service.cc:1467] Processing CreateTablet for tablet f1809a8a6a9f458bb5d9de9bcf7bd179 (DEFAULT_TABLE table=default.Table_be0171ec0e714233b40b175da1e82b4d [id=be73b2a1315a45baa1db6feb44a3843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.872126 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 182a3de861984b7d978f15a17275b6b0 (DEFAULT_TABLE table=default.Table_153effa513de4385b0ddfd1d561446ac [id=afed6df2d7b14d34b8274cc71e15c12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.872234 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e726aa52406d49d3b5c78f558755ac6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.872393 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a49d621e94c4eadb63be52e58c3fab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.872555 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec23d785839462b91c036705cac2a72 (DEFAULT_TABLE table=default.Table_b879654fd2304197a107fc833fd81e21 [id=c9ddd47c049f435eafb85d7bd033cc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.872869 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec23d785839462b91c036705cac2a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.873838 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48e1bae892a41caa69609dd89700939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.877574 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 88d16a40c7c24972861d582fb68d4da4 (DEFAULT_TABLE table=default.Table_8ab9abf5e45d49b1a3377c34813da069 [id=a4f8f81ef4de4f1589dd6102b92e44ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.877939 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d16a40c7c24972861d582fb68d4da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.878015 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 1893a7de2d52475db1baeb4e816b549b (DEFAULT_TABLE table=default.Table_9277b5f5bbf44d878345b1028267ba8a [id=274ffbb5e513489f85dcc6097c8cc444]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.878432 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1893a7de2d52475db1baeb4e816b549b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.881866 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a3de861984b7d978f15a17275b6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.882573 13242 log.cc:1196] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/da41ef03182445f1946584c2bf015139
I20240506 19:56:54.882982 13242 ts_tablet_manager.cc:1909] T da41ef03182445f1946584c2bf015139 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:54.885079 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec23d785839462b91c036705cac2a72 (DEFAULT_TABLE table=default.Table_b879654fd2304197a107fc833fd81e21 [id=c9ddd47c049f435eafb85d7bd033cc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.885604 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec23d785839462b91c036705cac2a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.871796 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 1893a7de2d52475db1baeb4e816b549b (DEFAULT_TABLE table=default.Table_9277b5f5bbf44d878345b1028267ba8a [id=274ffbb5e513489f85dcc6097c8cc444]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.886026 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1893a7de2d52475db1baeb4e816b549b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.889515 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1809a8a6a9f458bb5d9de9bcf7bd179. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.889972 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 88d16a40c7c24972861d582fb68d4da4 (DEFAULT_TABLE table=default.Table_8ab9abf5e45d49b1a3377c34813da069 [id=a4f8f81ef4de4f1589dd6102b92e44ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.890161 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 4a49d621e94c4eadb63be52e58c3fab5 (DEFAULT_TABLE table=default.Table_4dcb29fa5425483296ab644d95581803 [id=c8a04b6ee94a46c4995bbd1182157875]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.890376 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d16a40c7c24972861d582fb68d4da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.890647 13088 tablet_service.cc:1467] Processing CreateTablet for tablet f1809a8a6a9f458bb5d9de9bcf7bd179 (DEFAULT_TABLE table=default.Table_be0171ec0e714233b40b175da1e82b4d [id=be73b2a1315a45baa1db6feb44a3843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.891045 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1809a8a6a9f458bb5d9de9bcf7bd179. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.892153 14436 raft_consensus.cc:482] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.892295 14436 raft_consensus.cc:504] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.893011 14436 leader_election.cc:288] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.893939 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a49d621e94c4eadb63be52e58c3fab5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:54.897918 12783 leader_election.cc:341] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 67e8b8be953a4a5784f2d098fd6364e3
I20240506 19:56:54.898582 14398 raft_consensus.cc:482] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.898837 14398 raft_consensus.cc:504] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.899796 14402 raft_consensus.cc:482] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.899858 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 060cfeedba024cebb19ccccb49d4d582 (DEFAULT_TABLE table=default.Table_f9d1e5113608496eb4aad3218eb44a65 [id=f7188aea4c354dd4920e4fd94f6b5323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.899960 14402 raft_consensus.cc:504] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:54.900327 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060cfeedba024cebb19ccccb49d4d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.900411 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a8531a1793404699532295164a3895" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.900620 14402 leader_election.cc:288] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:54.901158 13058 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.901597 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 934ea08878b84c6c8e5ef7ca0d3fa65c (DEFAULT_TABLE table=default.Table_33e9d54058954ad9806125313b789621 [id=6cab7f5bc39a44e187c9098a05c2e4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.901708 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:54.901965 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934ea08878b84c6c8e5ef7ca0d3fa65c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:54.902302 13058 leader_election.cc:341] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:54.902964 14398 leader_election.cc:288] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:54.871527 12960 tablet_service.cc:1467] Processing CreateTablet for tablet a02b198026be469282f8dc0e82181973 (DEFAULT_TABLE table=default.Table_68c8729d5e134c10be8d88e246c4e00d [id=4537e3ced96b436a9e4d134427351495]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:54.907773 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a02b198026be469282f8dc0e82181973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.920231 13246 tablet_replica.cc:330] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.920464 13246 raft_consensus.cc:2229] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.920665 13246 raft_consensus.cc:2258] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.921218 13246 ts_tablet_manager.cc:1875] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.921739 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 997ecba7fbae41178a25bf7af4d152e1 (table default.Table_b666605912f04406bd8d879caa8c8b9b [id=85453d1f205a4fc5bcf2dd05063162f8]) successfully deleted
I20240506 19:56:54.922469 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a3de861984b7d978f15a17275b6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.923127 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934ea08878b84c6c8e5ef7ca0d3fa65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.923667 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e726aa52406d49d3b5c78f558755ac6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:54.929184 13246 ts_tablet_manager.cc:1888] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.929657 13246 log.cc:1196] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/452a83470585433e8def7174300d3a7f
I20240506 19:56:54.930085 13246 ts_tablet_manager.cc:1909] T 452a83470585433e8def7174300d3a7f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.930692 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 452a83470585433e8def7174300d3a7f (table default.Table_2e36b2f9be7f4af9a58215847e671f2f [id=3a1dae7697ec4d529d8bb162d1170b3a]) successfully deleted
I20240506 19:56:54.931006 13246 tablet_replica.cc:330] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.931229 13246 raft_consensus.cc:2229] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.931387 13246 raft_consensus.cc:2258] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.931946 13246 ts_tablet_manager.cc:1875] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.938834 13246 ts_tablet_manager.cc:1888] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.939191 13246 log.cc:1196] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:56:54.939829 13246 ts_tablet_manager.cc:1909] T 60d1f01c0edb4f6b8a621c1643ca1afb P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.940421 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 60d1f01c0edb4f6b8a621c1643ca1afb (table default.Table_200204bb39c0432081665c45b4aa9eae [id=4884fa87524f4bdebb7c9e61f25a48a6]) successfully deleted
I20240506 19:56:54.940961 13246 tablet_replica.cc:330] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.941190 13246 raft_consensus.cc:2229] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.941460 14436 raft_consensus.cc:482] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:54.941603 14436 raft_consensus.cc:504] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:54.942315 14436 leader_election.cc:288] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:54.943800 12783 leader_election.cc:341] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 4095f9fc28ff48f68064ec8318ff7b51
I20240506 19:56:54.944402 13246 raft_consensus.cc:2258] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.945065 13246 ts_tablet_manager.cc:1875] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.951771 13246 ts_tablet_manager.cc:1888] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.952004 13246 log.cc:1196] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:56:54.952430 13246 ts_tablet_manager.cc:1909] T da06187c83b94d03ae4ebb67a2873cf0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.953172 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet da06187c83b94d03ae4ebb67a2873cf0 (table default.Table_b261bf09f51646a0877f068d9eb2ab36 [id=034d601564f94db89efafdf7d3b35f05]) successfully deleted
I20240506 19:56:54.953588 13246 tablet_replica.cc:330] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.953855 13246 raft_consensus.cc:2229] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.954099 13246 raft_consensus.cc:2258] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.954715 13246 ts_tablet_manager.cc:1875] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.961036 13246 ts_tablet_manager.cc:1888] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.961272 13246 log.cc:1196] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/a1160ab972124630a9be7abeefb4aa56
I20240506 19:56:54.961750 13246 ts_tablet_manager.cc:1909] T a1160ab972124630a9be7abeefb4aa56 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.962468 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet a1160ab972124630a9be7abeefb4aa56 (table default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]) successfully deleted
I20240506 19:56:54.962954 13246 tablet_replica.cc:330] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.963218 13246 raft_consensus.cc:2229] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.963507 13246 raft_consensus.cc:2258] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:54.963812 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b2be78cbc8264f6e8d2e05137135161c because the tablet was not found. No further retry: Not found: Tablet not found: b2be78cbc8264f6e8d2e05137135161c
I20240506 19:56:54.964572 13246 ts_tablet_manager.cc:1875] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.972277 13246 ts_tablet_manager.cc:1888] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.972584 13246 log.cc:1196] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/116df9bd6ab04a30ba09861e765d7267
I20240506 19:56:54.973070 13246 ts_tablet_manager.cc:1909] T 116df9bd6ab04a30ba09861e765d7267 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.973788 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 116df9bd6ab04a30ba09861e765d7267 (table default.Table_72ce5aee893c4fd78aca451ac2b7b773 [id=db8bd7495bce4c518581981bca778450]) successfully deleted
W20240506 19:56:54.974510 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet edaaead0cf2e4133b8bdec62fb4770a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edaaead0cf2e4133b8bdec62fb4770a0 already in progress: creating tablet
I20240506 19:56:54.974927 13246 tablet_replica.cc:330] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.975288 13246 raft_consensus.cc:2229] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.975637 13246 raft_consensus.cc:2258] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.976577 13246 ts_tablet_manager.cc:1875] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.983104 13246 ts_tablet_manager.cc:1888] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:54.983405 13246 log.cc:1196] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/99f4861a01404663870ce91978c8af4f
I20240506 19:56:54.983937 13246 ts_tablet_manager.cc:1909] T 99f4861a01404663870ce91978c8af4f P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:54.984719 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 99f4861a01404663870ce91978c8af4f (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:54.985069 13246 tablet_replica.cc:330] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:54.985375 13246 raft_consensus.cc:2229] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:54.985632 13246 raft_consensus.cc:2258] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:54.986200 13246 ts_tablet_manager.cc:1875] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:54.992821 13246 ts_tablet_manager.cc:1888] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:54.993065 13246 log.cc:1196] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/85bc7cfd006f4d4ca95541dab6873d32
I20240506 19:56:54.993567 13246 ts_tablet_manager.cc:1909] T 85bc7cfd006f4d4ca95541dab6873d32 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.005324 13244 tablet_bootstrap.cc:492] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.005797 12816 tablet_service.cc:1467] Processing CreateTablet for tablet a02b198026be469282f8dc0e82181973 (DEFAULT_TABLE table=default.Table_68c8729d5e134c10be8d88e246c4e00d [id=4537e3ced96b436a9e4d134427351495]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.006328 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 934ea08878b84c6c8e5ef7ca0d3fa65c (DEFAULT_TABLE table=default.Table_33e9d54058954ad9806125313b789621 [id=6cab7f5bc39a44e187c9098a05c2e4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.006723 13244 tablet_bootstrap.cc:654] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.007357 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934ea08878b84c6c8e5ef7ca0d3fa65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.007683 14383 raft_consensus.cc:482] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.007815 14383 raft_consensus.cc:504] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.008394 14383 leader_election.cc:288] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:55.008909 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997ecba7fbae41178a25bf7af4d152e1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:55.009367 12921 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.013139 12825 tablet_service.cc:1467] Processing CreateTablet for tablet e726aa52406d49d3b5c78f558755ac6e (DEFAULT_TABLE table=default.Table_6c36d4fe94e7454280ad13d12a5ed6db [id=e354c012edc64f49bdc0ab0dde85d05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.013715 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 182a3de861984b7d978f15a17275b6b0 (DEFAULT_TABLE table=default.Table_153effa513de4385b0ddfd1d561446ac [id=afed6df2d7b14d34b8274cc71e15c12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.014057 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a3de861984b7d978f15a17275b6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.017645 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f41319b312d48c6bd921641086c7b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.017683 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 1893a7de2d52475db1baeb4e816b549b (DEFAULT_TABLE table=default.Table_9277b5f5bbf44d878345b1028267ba8a [id=274ffbb5e513489f85dcc6097c8cc444]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.018483 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1893a7de2d52475db1baeb4e816b549b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.020301 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41aa8aa4c9a4163b1eae92f11a74b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.024153 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 4a49d621e94c4eadb63be52e58c3fab5 (DEFAULT_TABLE table=default.Table_4dcb29fa5425483296ab644d95581803 [id=c8a04b6ee94a46c4995bbd1182157875]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.024610 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a49d621e94c4eadb63be52e58c3fab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.025756 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2441d75b9bc48409aa0022d6a30d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.029827 12827 tablet_service.cc:1467] Processing CreateTablet for tablet f1809a8a6a9f458bb5d9de9bcf7bd179 (DEFAULT_TABLE table=default.Table_be0171ec0e714233b40b175da1e82b4d [id=be73b2a1315a45baa1db6feb44a3843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.031435 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 879) took 3450 ms. Trace:
I20240506 19:56:55.029839 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a369e21e62444208be40d8c552b1626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.032037 12826 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec23d785839462b91c036705cac2a72 (DEFAULT_TABLE table=default.Table_b879654fd2304197a107fc833fd81e21 [id=c9ddd47c049f435eafb85d7bd033cc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.032356 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec23d785839462b91c036705cac2a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.032521 13244 tablet_bootstrap.cc:492] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.032703 13244 ts_tablet_manager.cc:1367] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.009s
I20240506 19:56:55.035495 13244 raft_consensus.cc:348] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.035708 13244 raft_consensus.cc:374] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.035763 13244 raft_consensus.cc:729] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.036015 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f643b5e89e470aa35f258109bbc22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.036207 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dfe8f732074180a159027013cec014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.035960 13244 consensus_queue.cc:260] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.036674 13244 ts_tablet_manager.cc:1398] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:55.036940 13244 tablet_bootstrap.cc:492] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.038479 13244 tablet_bootstrap.cc:654] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.041474 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5024b8379db74d278f91894fad290691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.031543 13242 rpcz_store.cc:276] 0506 19:56:51.581171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.581237 (+ 66us) service_pool.cc:225] Handling call
0506 19:56:54.586543 (+3005306us) ts_tablet_manager.cc:1120] Deleting tablet da41ef03182445f1946584c2bf015139
0506 19:56:54.586556 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:54.611464 (+ 24908us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:54.869516 (+258052us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.031412 (+161896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005064}
I20240506 19:56:55.041934 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 880) took 3441 ms. Trace:
I20240506 19:56:55.042016 13242 rpcz_store.cc:276] 0506 19:56:51.600468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.600561 (+ 93us) service_pool.cc:225] Handling call
0506 19:56:55.041797 (+3441236us) ts_tablet_manager.cc:1120] Deleting tablet d43da1d12ce647dcbcd31263932b2f54
0506 19:56:55.041807 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.041915 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3441020}
I20240506 19:56:55.042316 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 881) took 3431 ms. Trace:
I20240506 19:56:55.042376 13242 rpcz_store.cc:276] 0506 19:56:51.610396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.610465 (+ 69us) service_pool.cc:225] Handling call
0506 19:56:55.042226 (+3431761us) ts_tablet_manager.cc:1120] Deleting tablet 691e312510a440898a58bec43ab26e8f
0506 19:56:55.042234 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.042304 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3431511}
I20240506 19:56:55.042640 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 882) took 3385 ms. Trace:
I20240506 19:56:55.042726 13242 rpcz_store.cc:276] 0506 19:56:51.657075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.657129 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:55.042542 (+3385413us) ts_tablet_manager.cc:1120] Deleting tablet af2741336d5a4be1b5306319fc20cdc6
0506 19:56:55.042549 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.042620 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3385169}
I20240506 19:56:55.042971 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 883) took 3383 ms. Trace:
I20240506 19:56:55.043027 13242 rpcz_store.cc:276] 0506 19:56:51.659157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.659210 (+ 53us) service_pool.cc:225] Handling call
0506 19:56:55.042892 (+3383682us) ts_tablet_manager.cc:1120] Deleting tablet e6d37bb1c2144c4fa71aab6113b9f06e
0506 19:56:55.042899 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.042959 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3383509}
I20240506 19:56:55.043282 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 885) took 3349 ms. Trace:
I20240506 19:56:55.043341 13242 rpcz_store.cc:276] 0506 19:56:51.693591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.693658 (+ 67us) service_pool.cc:225] Handling call
0506 19:56:55.043200 (+3349542us) ts_tablet_manager.cc:1120] Deleting tablet c8eeaa71ac61424796c6ffdd993de9db
0506 19:56:55.043208 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.043269 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3349319}
I20240506 19:56:55.043601 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 884) took 3350 ms. Trace:
I20240506 19:56:55.043669 13242 rpcz_store.cc:276] 0506 19:56:51.693476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.693693 (+ 217us) service_pool.cc:225] Handling call
0506 19:56:55.043523 (+3349830us) ts_tablet_manager.cc:1120] Deleting tablet c1a5a8dc9a2a4856b8b8f2aaac187c36
0506 19:56:55.043530 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.043590 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3349620}
I20240506 19:56:55.043874 13242 tablet_replica.cc:330] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.044024 13242 raft_consensus.cc:2229] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.044153 13242 raft_consensus.cc:2258] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.044719 13242 ts_tablet_manager.cc:1875] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.045444 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7380f259c68d4d7c89938a548533b354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.045763 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041612b3636a49db9e303877964a31ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.045946 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.049425 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe159e09e324e4bac409d2d6269292f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.049654 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet da41ef03182445f1946584c2bf015139 (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
W20240506 19:56:55.049974 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d43da1d12ce647dcbcd31263932b2f54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43da1d12ce647dcbcd31263932b2f54 already in progress: creating tablet
W20240506 19:56:55.050177 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 691e312510a440898a58bec43ab26e8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691e312510a440898a58bec43ab26e8f already in progress: creating tablet
W20240506 19:56:55.050351 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet af2741336d5a4be1b5306319fc20cdc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af2741336d5a4be1b5306319fc20cdc6 already in progress: creating tablet
W20240506 19:56:55.050501 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet e6d37bb1c2144c4fa71aab6113b9f06e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6d37bb1c2144c4fa71aab6113b9f06e already in progress: creating tablet
W20240506 19:56:55.050668 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c8eeaa71ac61424796c6ffdd993de9db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8eeaa71ac61424796c6ffdd993de9db already in progress: creating tablet
W20240506 19:56:55.050837 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1a5a8dc9a2a4856b8b8f2aaac187c36 already in progress: creating tablet
I20240506 19:56:55.052515 13244 tablet_bootstrap.cc:492] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.052649 13244 ts_tablet_manager.cc:1367] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240506 19:56:55.053411 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98d8aed6bc8460b9be6b8bfd00373d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.053359 13244 raft_consensus.cc:348] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.053529 13244 raft_consensus.cc:374] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.053582 13244 raft_consensus.cc:729] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.053771 13244 consensus_queue.cc:260] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.054240 13244 ts_tablet_manager.cc:1398] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:55.057498 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060cfeedba024cebb19ccccb49d4d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.061421 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9fa00ac9f645359b51384fba7179bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.062372 14436 raft_consensus.cc:482] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.062558 14436 raft_consensus.cc:504] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.063349 14436 leader_election.cc:288] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:55.064204 12783 leader_election.cc:341] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bafd3c4706f74e7da36a8f3538760f43
I20240506 19:56:55.069520 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a02b198026be469282f8dc0e82181973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.073798 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 88d16a40c7c24972861d582fb68d4da4 (DEFAULT_TABLE table=default.Table_8ab9abf5e45d49b1a3377c34813da069 [id=a4f8f81ef4de4f1589dd6102b92e44ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.074290 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d16a40c7c24972861d582fb68d4da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.074924 13244 tablet_bootstrap.cc:492] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.076505 13244 tablet_bootstrap.cc:654] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.081403 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e726aa52406d49d3b5c78f558755ac6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.083619 14436 raft_consensus.cc:482] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.083773 14436 raft_consensus.cc:504] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.084414 14436 leader_election.cc:288] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.085258 14436 raft_consensus.cc:482] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.085410 14436 raft_consensus.cc:504] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.085913 14436 leader_election.cc:288] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.087157 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1809a8a6a9f458bb5d9de9bcf7bd179. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:55.087234 12783 leader_election.cc:341] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 03df4632adfa4b0ab999aeaa5bb4e1f4
I20240506 19:56:55.087723 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:55.091243 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 85bc7cfd006f4d4ca95541dab6873d32 (table default.Table_d909ad6f09e24952918c9471609c9d27 [id=2ebd467d8ac3476192b5634ac512aa47]) successfully deleted
I20240506 19:56:55.094660 13242 ts_tablet_manager.cc:1888] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.094794 13242 log.cc:1196] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/d8f3e7df09f347bfa5cad4de34dbda13
I20240506 19:56:55.095062 13242 ts_tablet_manager.cc:1909] T d8f3e7df09f347bfa5cad4de34dbda13 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:55.095363 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 886) took 3368 ms. Trace:
I20240506 19:56:55.095453 13242 rpcz_store.cc:276] 0506 19:56:51.726508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.726579 (+ 71us) service_pool.cc:225] Handling call
0506 19:56:55.043836 (+3317257us) ts_tablet_manager.cc:1120] Deleting tablet d8f3e7df09f347bfa5cad4de34dbda13
0506 19:56:55.043842 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.090859 (+ 47017us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.094317 (+ 3458us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.095343 (+ 1026us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3317002}
I20240506 19:56:55.095924 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet d8f3e7df09f347bfa5cad4de34dbda13 (table default.Table_d5bb6a8c548e46ecb59035f49816ad52 [id=70785b5175b34eb68dc3fcba2255645b]) successfully deleted
I20240506 19:56:55.096580 13244 tablet_bootstrap.cc:492] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.096712 13244 ts_tablet_manager.cc:1367] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20240506 19:56:55.098028 13244 raft_consensus.cc:348] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.098291 13244 raft_consensus.cc:374] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.098366 13244 raft_consensus.cc:729] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.098619 13244 consensus_queue.cc:260] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.098723 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 889) took 3343 ms. Trace:
I20240506 19:56:55.098930 13242 rpcz_store.cc:276] 0506 19:56:51.755390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.755447 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:55.098533 (+3343086us) ts_tablet_manager.cc:1120] Deleting tablet 15ed0c7b4569490b91c54d5bdb187e49
0506 19:56:55.098545 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.098701 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342851}
W20240506 19:56:55.099575 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 15ed0c7b4569490b91c54d5bdb187e49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ed0c7b4569490b91c54d5bdb187e49 already in progress: creating tablet
I20240506 19:56:55.100502 13244 ts_tablet_manager.cc:1398] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:55.100816 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 890) took 3332 ms. Trace:
I20240506 19:56:55.100960 13242 rpcz_store.cc:276] 0506 19:56:51.768659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.769210 (+ 551us) service_pool.cc:225] Handling call
0506 19:56:55.100081 (+3330871us) ts_tablet_manager.cc:1120] Deleting tablet 6a7715dad8184493b085d891428ad44c
0506 19:56:55.100663 (+ 582us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.100794 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3330636}
W20240506 19:56:55.101693 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 6a7715dad8184493b085d891428ad44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a7715dad8184493b085d891428ad44c already in progress: creating tablet
I20240506 19:56:55.101770 13244 tablet_bootstrap.cc:492] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.102192 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 895) took 3236 ms. Trace:
I20240506 19:56:55.102440 13242 rpcz_store.cc:276] 0506 19:56:51.865528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:51.865580 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:55.102084 (+3236504us) ts_tablet_manager.cc:1120] Deleting tablet 44ce8749846847108edf5f554139cbd9
0506 19:56:55.102093 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.102174 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3236301}
W20240506 19:56:55.103753 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 44ce8749846847108edf5f554139cbd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ce8749846847108edf5f554139cbd9 already in progress: creating tablet
I20240506 19:56:55.104212 13242 tablet_replica.cc:330] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.104450 13242 raft_consensus.cc:2229] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.104552 13246 tablet_replica.cc:330] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.104761 13242 raft_consensus.cc:2258] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.106429 13242 ts_tablet_manager.cc:1875] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.108747 12824 tablet_service.cc:1467] Processing CreateTablet for tablet a48e1bae892a41caa69609dd89700939 (DEFAULT_TABLE table=default.Table_c1b76d968cf047ab833ec4ec4af83697 [id=2c499becf4f44f84bc6d33b9d92d8791]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.109161 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48e1bae892a41caa69609dd89700939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.109911 13244 tablet_bootstrap.cc:654] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.113126 13242 ts_tablet_manager.cc:1888] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.113253 13242 log.cc:1196] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/67e8b8be953a4a5784f2d098fd6364e3
I20240506 19:56:55.117735 13245 ts_tablet_manager.cc:1398] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.339s user 0.002s sys 0.000s
I20240506 19:56:55.118083 13246 raft_consensus.cc:2229] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.118312 13246 raft_consensus.cc:2258] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.118914 13246 ts_tablet_manager.cc:1875] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.119021 13244 tablet_bootstrap.cc:492] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.122916 13244 ts_tablet_manager.cc:1367] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240506 19:56:55.123104 13242 ts_tablet_manager.cc:1909] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:55.123440 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 898) took 3138 ms. Trace:
I20240506 19:56:55.123543 13242 rpcz_store.cc:276] 0506 19:56:51.984987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.001794 (+ 16807us) service_pool.cc:225] Handling call
0506 19:56:55.104165 (+3102371us) ts_tablet_manager.cc:1120] Deleting tablet 67e8b8be953a4a5784f2d098fd6364e3
0506 19:56:55.104175 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.109921 (+ 5746us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.113023 (+ 3102us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.123419 (+ 10396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3102154}
I20240506 19:56:55.124176 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 67e8b8be953a4a5784f2d098fd6364e3 (table default.Table_cce4d16b2e3744bc9d2be183d5d1dbeb [id=7a2c513cd8e74817b4a391c9c7932085]) successfully deleted
I20240506 19:56:55.123770 13244 raft_consensus.cc:348] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.124915 13244 raft_consensus.cc:374] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.125046 13244 raft_consensus.cc:729] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.123986 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 899) took 3133 ms. Trace:
I20240506 19:56:55.125429 13244 consensus_queue.cc:260] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.125376 13242 rpcz_store.cc:276] 0506 19:56:51.990733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.002046 (+ 11313us) service_pool.cc:225] Handling call
0506 19:56:55.123851 (+3121805us) ts_tablet_manager.cc:1120] Deleting tablet 8a9b92795a7742caa2ee99ab7ca138bf
0506 19:56:55.123863 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.123971 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3121669}
I20240506 19:56:55.125875 13242 tablet_replica.cc:330] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.126063 13242 raft_consensus.cc:2229] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:55.126089 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8a9b92795a7742caa2ee99ab7ca138bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a9b92795a7742caa2ee99ab7ca138bf already in progress: creating tablet
I20240506 19:56:55.126204 13242 raft_consensus.cc:2258] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.126729 13242 ts_tablet_manager.cc:1875] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.127234 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:55.127291 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a8531a1793404699532295164a3895" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:55.127835 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6e733b6efd4041abf465c7bb788ef5" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:55.128588 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35231d3f1bb4348aa7af3c71b809cd9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:55.129065 13244 ts_tablet_manager.cc:1398] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:55.129477 13244 tablet_bootstrap.cc:492] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.133210 13244 tablet_bootstrap.cc:654] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.137501 13245 tablet_bootstrap.cc:492] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:55.144984 13245 tablet_bootstrap.cc:654] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.145968 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 01da81cb9ee840ed9e6640af305eaecc
I20240506 19:56:55.146414 13244 tablet_bootstrap.cc:492] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.146456 12514 catalog_manager.cc:6554] ReplaceTablet: tablet 01da81cb9ee840ed9e6640af305eaecc deleted and replaced by tablet a47b76562ce04a72b73b5055afb04f0c
I20240506 19:56:55.146548 13244 ts_tablet_manager.cc:1367] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240506 19:56:55.147507 13244 raft_consensus.cc:348] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.147843 13244 raft_consensus.cc:374] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.147894 13244 raft_consensus.cc:729] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.147925 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0fb9c05d703f417780e77359e19e542e
I20240506 19:56:55.148113 13244 consensus_queue.cc:260] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.150364 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 01da81cb9ee840ed9e6640af305eaecc with delete_type TABLET_DATA_DELETED (replaced by tablet a47b76562ce04a72b73b5055afb04f0c) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.150468 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb9c05d703f417780e77359e19e542e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.151522 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 01da81cb9ee840ed9e6640af305eaecc with delete_type TABLET_DATA_DELETED (replaced by tablet a47b76562ce04a72b73b5055afb04f0c) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.151618 13244 ts_tablet_manager.cc:1398] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240506 19:56:55.151863 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb9c05d703f417780e77359e19e542e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.151914 13244 tablet_bootstrap.cc:492] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.152537 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb9c05d703f417780e77359e19e542e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.153605 13244 tablet_bootstrap.cc:654] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.154160 14436 raft_consensus.cc:482] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.154282 14436 raft_consensus.cc:504] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.154944 14436 leader_election.cc:288] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:55.156060 12783 leader_election.cc:341] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 17fb091762144f6591152680a6acd29e
I20240506 19:56:55.163801 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 01da81cb9ee840ed9e6640af305eaecc with delete_type TABLET_DATA_DELETED (replaced by tablet a47b76562ce04a72b73b5055afb04f0c) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.166842 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 141bca5c816a494b83704a915f0edfd1
I20240506 19:56:55.167380 12922 leader_election.cc:302] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.167428 12520 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 1212) took 1244 ms. Trace:
I20240506 19:56:55.167815 14383 raft_consensus.cc:2781] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:55.167929 14383 raft_consensus.cc:686] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:55.167543 12520 rpcz_store.cc:276] 0506 19:56:53.923288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.262048 (+338760us) service_pool.cc:225] Handling call
0506 19:56:54.262919 (+ 871us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:54.263121 (+ 202us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:54.263208 (+ 87us) catalog_manager.cc:2758] Locking tablets
0506 19:56:54.623144 (+359936us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:55.166598 (+543454us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:55.166709 (+ 111us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:55.166759 (+ 50us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:55.167389 (+ 630us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:54.623826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:54.623916 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:54.623921 (+ 5us) write_op.cc:435] Acquired schema lock
0506 19:56:54.623923 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:54.624018 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:54.624029 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:54.624034 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:54.624117 (+ 83us) tablet.cc:699] Row locks acquired
0506 19:56:54.624120 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:54.624148 (+ 28us) write_op.cc:270] Start()
0506 19:56:54.624178 (+ 30us) write_op.cc:276] Timestamp: P: 1715025414624198 usec, L: 0
0506 19:56:54.624181 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:54.624325 (+ 144us) log.cc:842] Serialized 731 byte log entry
0506 19:56:55.165368 (+541043us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.165527 (+ 159us) write_op.cc:301] APPLY: starting
0506 19:56:55.165545 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.165603 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.165635 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.165709 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.165749 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:55.165757 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:55.165846 (+ 89us) log.cc:842] Serialized 29 byte log entry
0506 19:56:55.165915 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.165937 (+ 22us) write_op.cc:454] Released schema lock
0506 19:56:55.165944 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.165999 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.166002 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":528,"apply.run_wall_time_us":528,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":559,"replication_time_us":541159}]]}
I20240506 19:56:55.168176 14383 consensus_queue.cc:237] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.168664 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 182a3de861984b7d978f15a17275b6b0
I20240506 19:56:55.170786 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 141bca5c816a494b83704a915f0edfd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.172020 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 182a3de861984b7d978f15a17275b6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.172199 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 182a3de861984b7d978f15a17275b6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.171931 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 141bca5c816a494b83704a915f0edfd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.173614 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 1980a68925be4d3da1078c78d32da866
I20240506 19:56:55.174279 12513 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 1211) took 1252 ms. Trace:
I20240506 19:56:55.174412 12513 rpcz_store.cc:276] 0506 19:56:53.922161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.222081 (+299920us) service_pool.cc:225] Handling call
0506 19:56:54.225405 (+ 3324us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:54.243760 (+ 18355us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:54.243862 (+ 102us) catalog_manager.cc:2758] Locking tablets
0506 19:56:54.626576 (+382714us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:55.173419 (+546843us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:55.173509 (+ 90us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:55.173555 (+ 46us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:55.174205 (+ 650us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:54.627192 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:54.627280 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:54.627285 (+ 5us) write_op.cc:435] Acquired schema lock
0506 19:56:54.627288 (+ 3us) tablet.cc:660] Decoding operations
0506 19:56:54.627398 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:54.627407 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:54.627411 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:54.627485 (+ 74us) tablet.cc:699] Row locks acquired
0506 19:56:54.627488 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:54.627516 (+ 28us) write_op.cc:270] Start()
0506 19:56:54.627539 (+ 23us) write_op.cc:276] Timestamp: P: 1715025414627568 usec, L: 0
0506 19:56:54.627542 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:54.627680 (+ 138us) log.cc:842] Serialized 731 byte log entry
0506 19:56:55.172464 (+544784us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.172604 (+ 140us) write_op.cc:301] APPLY: starting
0506 19:56:55.172631 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.172699 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.172754 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.172864 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.172915 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:55.172928 (+ 13us) write_op.cc:312] APPLY: finished
0506 19:56:55.173018 (+ 90us) log.cc:842] Serialized 29 byte log entry
0506 19:56:55.173100 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.173124 (+ 24us) write_op.cc:454] Released schema lock
0506 19:56:55.173131 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.173185 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.173188 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":638,"apply.run_wall_time_us":639,"num_ops":2,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":542,"prepare.run_wall_time_us":543,"replication_time_us":544890}]]}
W20240506 19:56:55.175866 12507 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 92329da0f9ec4d2e982efe5e9b94c5d6 (table default.Table_8c32c6726a7f4aa2867ec4f2594083d5 [id=bc316585ccd5468cba6c1016998d5d21], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92329da0f9ec4d2e982efe5e9b94c5d6 consensus state has no leader
I20240506 19:56:55.175899 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 141bca5c816a494b83704a915f0edfd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.176863 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 1980a68925be4d3da1078c78d32da866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.177078 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 1980a68925be4d3da1078c78d32da866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.178555 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2919538141834fb6a83259e29288acdd
I20240506 19:56:55.179116 12522 catalog_manager.cc:6554] ReplaceTablet: tablet 2919538141834fb6a83259e29288acdd deleted and replaced by tablet 1cc57eb8ac864bfa803908412530e7a3
I20240506 19:56:55.177202 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 1980a68925be4d3da1078c78d32da866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.180316 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 2919538141834fb6a83259e29288acdd with delete_type TABLET_DATA_DELETED (replaced by tablet 1cc57eb8ac864bfa803908412530e7a3) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.181380 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2919538141834fb6a83259e29288acdd with delete_type TABLET_DATA_DELETED (replaced by tablet 1cc57eb8ac864bfa803908412530e7a3) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.182103 13249 tablet_bootstrap.cc:492] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.182448 13244 tablet_bootstrap.cc:492] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.182605 13244 ts_tablet_manager.cc:1367] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20240506 19:56:55.181344 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet 2919538141834fb6a83259e29288acdd with delete_type TABLET_DATA_DELETED (replaced by tablet 1cc57eb8ac864bfa803908412530e7a3) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.185484 13249 ts_tablet_manager.cc:1367] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.343s user 0.054s sys 0.047s
I20240506 19:56:55.185844 13244 raft_consensus.cc:348] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.186095 13244 raft_consensus.cc:374] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.186187 13244 raft_consensus.cc:729] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.184022 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 182a3de861984b7d978f15a17275b6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:54 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.186460 13244 consensus_queue.cc:260] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:55.186861 12922 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 234e4f3439e84b5fbb2f7bab1dbd0822
I20240506 19:56:55.187147 13244 ts_tablet_manager.cc:1398] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:55.189234 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:55.189471 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:55.189651 12518 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:55.190006 12922 leader_election.cc:302] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.191469 12516 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97] alter complete (version 2)
I20240506 19:56:55.186959 13249 raft_consensus.cc:348] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.193459 14459 raft_consensus.cc:2726] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.193642 13249 raft_consensus.cc:374] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.193732 13249 raft_consensus.cc:729] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.193944 13249 consensus_queue.cc:260] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.194581 13249 ts_tablet_manager.cc:1398] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:55.194878 13249 tablet_bootstrap.cc:492] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.196801 13249 tablet_bootstrap.cc:654] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.198848 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:55.198870 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 01da81cb9ee840ed9e6640af305eaecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01da81cb9ee840ed9e6640af305eaecc already in progress: creating tablet
I20240506 19:56:55.199146 14557 tablet_replica.cc:330] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
W20240506 19:56:55.199229 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0fb9c05d703f417780e77359e19e542e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fb9c05d703f417780e77359e19e542e already in progress: creating tablet
I20240506 19:56:55.199374 14557 raft_consensus.cc:2229] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
W20240506 19:56:55.199442 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 182a3de861984b7d978f15a17275b6b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182a3de861984b7d978f15a17275b6b0 already in progress: creating tablet
W20240506 19:56:55.199669 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 141bca5c816a494b83704a915f0edfd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 141bca5c816a494b83704a915f0edfd1 already in progress: creating tablet
I20240506 19:56:55.198642 13242 ts_tablet_manager.cc:1888] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.199824 13242 log.cc:1196] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/cd57563cc6c140c184d1725a2da6a475
W20240506 19:56:55.199851 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1980a68925be4d3da1078c78d32da866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1980a68925be4d3da1078c78d32da866 already in progress: creating tablet
W20240506 19:56:55.200102 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2919538141834fb6a83259e29288acdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2919538141834fb6a83259e29288acdd already in progress: creating tablet
I20240506 19:56:55.200201 13242 ts_tablet_manager.cc:1909] T cd57563cc6c140c184d1725a2da6a475 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:55.202309 12922 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d7be9788df8e47d285bd932d078c6a8b
I20240506 19:56:55.202423 12922 leader_election.cc:302] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.202663 14557 raft_consensus.cc:2258] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.202957 13244 tablet_bootstrap.cc:492] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.203634 14557 ts_tablet_manager.cc:1875] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.204675 13244 tablet_bootstrap.cc:654] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.205030 12844 raft_consensus.cc:3037] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:55.205431 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.205637 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.205788 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.205926 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.206075 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.206101 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 04738def56324a368f8182543107e32e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.206238 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:55.206823 12519 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.194:59149 (request call id 134) took 1245 ms (client timeout 1000 ms). Trace:
I20240506 19:56:55.207249 12847 raft_consensus.cc:3037] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:55.209218 13246 ts_tablet_manager.cc:1888] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.209350 13246 log.cc:1196] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/2ff3435c11854649b7daf74f3d8c33d7
I20240506 19:56:55.209648 13246 ts_tablet_manager.cc:1909] T 2ff3435c11854649b7daf74f3d8c33d7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.210098 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet d9443cdbaa3c40299b66d216673c4b1a with delete_type TABLET_DATA_DELETED (replaced by tablet ca23d6f0a9e0445cbd4d0e708c8de2d3) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.210325 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.210415 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc with delete_type TABLET_DATA_DELETED (replaced by tablet efc41afd84a74652a82cc639bf562d99) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.210474 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 17da01620b8b41008421b977455e0c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.210546 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7c61244620487bbbac26dcdbf8a38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.210611 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.210772 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3268094cbf481698cb373c1bf6dfba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.212020 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.212287 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet bece304e0ba048ecbb2d92aa71fdf707 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.212468 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.212613 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b9731a473449b19a9e95653a9d0830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.212777 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
W20240506 19:56:55.213553 12921 consensus_peers.cc:485] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:55.206936 12519 rpcz_store.cc:269] 0506 19:56:53.963048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.182019 (+218971us) service_pool.cc:225] Handling call
0506 19:56:55.206764 (+1024745us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:54.649277 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:54.649484 (+ 207us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:54.649492 (+ 8us) write_op.cc:435] Acquired schema lock
0506 19:56:54.649494 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:54.649827 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:54.649869 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:54.649876 (+ 7us) tablet.cc:683] Acquiring locks for 5 operations
0506 19:56:54.650150 (+ 274us) tablet.cc:699] Row locks acquired
0506 19:56:54.650153 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:54.650227 (+ 74us) write_op.cc:270] Start()
0506 19:56:54.650294 (+ 67us) write_op.cc:276] Timestamp: P: 1715025414650277 usec, L: 0
0506 19:56:54.650298 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:54.650598 (+ 300us) log.cc:842] Serialized 2054 byte log entry
0506 19:56:55.182937 (+532339us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.183007 (+ 70us) write_op.cc:301] APPLY: starting
0506 19:56:55.183021 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.183077 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.183102 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.183205 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.183240 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0506 19:56:55.183246 (+ 6us) write_op.cc:312] APPLY: finished
0506 19:56:55.183319 (+ 73us) log.cc:842] Serialized 47 byte log entry
0506 19:56:55.183543 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.183630 (+ 87us) write_op.cc:454] Released schema lock
0506 19:56:55.183637 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":33,"num_ops":5,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":1147,"prepare.run_wall_time_us":1863,"replication_time_us":532619}]]}
I20240506 19:56:55.217077 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.217406 13245 tablet_bootstrap.cc:492] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:55.217784 13245 ts_tablet_manager.cc:1367] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20240506 19:56:55.218020 14460 raft_consensus.cc:2726] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.221494 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6c36d4fe94e7454280ad13d12a5ed6db" } new_table_name: "default.Table_4467002fce564e2fb50a2b9755805518" modify_external_catalogs: true
I20240506 19:56:55.221910 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 900) took 3212 ms. Trace:
I20240506 19:56:55.222537 13242 rpcz_store.cc:276] 0506 19:56:52.009161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.009215 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:55.125827 (+3116612us) ts_tablet_manager.cc:1120] Deleting tablet cd57563cc6c140c184d1725a2da6a475
0506 19:56:55.125837 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.137941 (+ 12104us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.198485 (+ 60544us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.221878 (+ 23393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3116382}
I20240506 19:56:55.223053 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_82ad0ca5d8e2425b98dc9d534a3436cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.223637 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82ad0ca5d8e2425b98dc9d534a3436cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.223960 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 903) took 3138 ms. Trace:
I20240506 19:56:55.224036 13242 rpcz_store.cc:276] 0506 19:56:52.085451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.086243 (+ 792us) service_pool.cc:225] Handling call
0506 19:56:55.223803 (+3137560us) ts_tablet_manager.cc:1120] Deleting tablet 35198d5786c14de89c9751a0372ba91d
0506 19:56:55.223817 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.223931 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3137302}
I20240506 19:56:55.224411 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 905) took 3084 ms. Trace:
I20240506 19:56:55.224472 13242 rpcz_store.cc:276] 0506 19:56:52.140126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.140190 (+ 64us) service_pool.cc:225] Handling call
0506 19:56:55.224296 (+3084106us) ts_tablet_manager.cc:1120] Deleting tablet 910d1465df784dfba6d78ff7fd9f2722
0506 19:56:55.224316 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.224395 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3083917}
I20240506 19:56:55.224745 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 907) took 2881 ms. Trace:
I20240506 19:56:55.224776 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet cd57563cc6c140c184d1725a2da6a475 (table default.Table_ff179844ac74496183bb76ddcabbcc75 [id=6a6084dba9e54d238890a1c9be17fdec]) successfully deleted
I20240506 19:56:55.224803 13242 rpcz_store.cc:276] 0506 19:56:52.343025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.343100 (+ 75us) service_pool.cc:225] Handling call
0506 19:56:55.224659 (+2881559us) ts_tablet_manager.cc:1120] Deleting tablet 5fe8d4891c484627ac729971ff7aad3e
0506 19:56:55.224667 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.224733 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2881295}
W20240506 19:56:55.225077 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 35198d5786c14de89c9751a0372ba91d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35198d5786c14de89c9751a0372ba91d already in progress: creating tablet
W20240506 19:56:55.225800 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 910d1465df784dfba6d78ff7fd9f2722 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 910d1465df784dfba6d78ff7fd9f2722 already in progress: creating tablet
W20240506 19:56:55.226071 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5fe8d4891c484627ac729971ff7aad3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fe8d4891c484627ac729971ff7aad3e already in progress: creating tablet
I20240506 19:56:55.227567 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_829c185590b0466f86672ee2ce1d349f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.228068 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_829c185590b0466f86672ee2ce1d349f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.229423 13242 tablet_replica.cc:330] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.229363 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5c27619f39744d3aa10bfc80a3397da5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.229724 13242 raft_consensus.cc:2229] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:55.229863 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c27619f39744d3aa10bfc80a3397da5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.229938 13242 raft_consensus.cc:2258] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.230659 13242 ts_tablet_manager.cc:1875] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.231353 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d0df8358053749a49c25b73041023ac1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.231873 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0df8358053749a49c25b73041023ac1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.233728 13245 raft_consensus.cc:348] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.234118 13245 raft_consensus.cc:374] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.234187 13245 raft_consensus.cc:729] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.234423 13245 consensus_queue.cc:260] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.235131 13245 ts_tablet_manager.cc:1398] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240506 19:56:55.240020 13057 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.239951 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2a09a031cefc4757938ad6047310bc72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.240151 13057 leader_election.cc:302] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.240414 13249 tablet_bootstrap.cc:492] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.240547 13249 ts_tablet_manager.cc:1367] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20240506 19:56:55.240525 14398 raft_consensus.cc:2726] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.240577 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a09a031cefc4757938ad6047310bc72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.240777 13244 tablet_bootstrap.cc:492] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.240936 13244 ts_tablet_manager.cc:1367] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.000s
W20240506 19:56:55.241262 13057 leader_election.cc:341] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.241387 13057 leader_election.cc:302] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.241356 13249 raft_consensus.cc:348] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.241549 13249 raft_consensus.cc:374] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.241601 13249 raft_consensus.cc:729] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.241709 14398 raft_consensus.cc:2726] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.241688 13244 raft_consensus.cc:348] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.241868 13244 raft_consensus.cc:374] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.241935 13244 raft_consensus.cc:729] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.242163 13244 consensus_queue.cc:260] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:55.242427 12781 leader_election.cc:341] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 67e8b8be953a4a5784f2d098fd6364e3
I20240506 19:56:55.242539 12781 leader_election.cc:302] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:55.242712 12781 raft_consensus.cc:2670] T 67e8b8be953a4a5784f2d098fd6364e3 P ef18e88b87064ed4b6c20f9325ebcc33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:55.242909 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a711b3a1487b4379a9e0eb9dacf05c1d" } modify_external_catalogs: true
I20240506 19:56:55.243095 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a711b3a1487b4379a9e0eb9dacf05c1d" } modify_external_catalogs: true
I20240506 19:56:55.243964 14474 tablet.cc:1722] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240506 19:56:55.244510 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ed6da25f3b94b26959e66e513088c16 consensus state has no leader
W20240506 19:56:55.249660 14489 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 18239770441020628405) tablet 00000000000000000000000000000000 cur holder 0x617000025380
I20240506 19:56:55.242723 13244 ts_tablet_manager.cc:1398] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:55.254856 13244 tablet_bootstrap.cc:492] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.241796 13249 consensus_queue.cc:260] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.257937 13249 ts_tablet_manager.cc:1398] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240506 19:56:55.258358 13249 tablet_bootstrap.cc:492] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.260876 13244 tablet_bootstrap.cc:654] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.262861 13249 tablet_bootstrap.cc:654] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.265610 13242 ts_tablet_manager.cc:1888] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.265766 13242 log.cc:1196] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2b5a46b91cb742a994496648fbf87aa4
I20240506 19:56:55.266119 13242 ts_tablet_manager.cc:1909] T 2b5a46b91cb742a994496648fbf87aa4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:55.260018 14557 ts_tablet_manager.cc:1888] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:55.266340 14557 log.cc:1196] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/99f4861a01404663870ce91978c8af4f
I20240506 19:56:55.266793 14557 ts_tablet_manager.cc:1909] T 99f4861a01404663870ce91978c8af4f P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.268296 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 906) took 2926 ms. Trace:
I20240506 19:56:55.268445 13242 rpcz_store.cc:276] 0506 19:56:52.341613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.350205 (+ 8592us) service_pool.cc:225] Handling call
0506 19:56:55.224970 (+2874765us) ts_tablet_manager.cc:1120] Deleting tablet 2b5a46b91cb742a994496648fbf87aa4
0506 19:56:55.229356 (+ 4386us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.261819 (+ 32463us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.265488 (+ 3669us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.268260 (+ 2772us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2874487}
I20240506 19:56:55.268976 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2b5a46b91cb742a994496648fbf87aa4 (table default.Table_3a6afcad38024709a1f104fa5bbadcff [id=2249bb6d1f824bacbd5b5b1c86e58c48]) successfully deleted
I20240506 19:56:55.269376 13242 tablet_replica.cc:330] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.269613 13242 raft_consensus.cc:2229] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.269790 13242 raft_consensus.cc:2258] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.270118 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 99f4861a01404663870ce91978c8af4f (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
W20240506 19:56:55.270571 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 971db6ac24d54efca95695ed1c90d4ed because the tablet was not found. No further retry: Not found: Tablet not found: 971db6ac24d54efca95695ed1c90d4ed
W20240506 19:56:55.270977 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 8eb9296bb1104b4cbe279eecd3683796 because the tablet was not found. No further retry: Not found: Tablet not found: 8eb9296bb1104b4cbe279eecd3683796
W20240506 19:56:55.271337 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 472c63d8995346d78633154a8f0d4c2b because the tablet was not found. No further retry: Not found: Tablet not found: 472c63d8995346d78633154a8f0d4c2b
I20240506 19:56:55.272395 13242 ts_tablet_manager.cc:1875] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.271036 14557 tablet_replica.cc:330] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.272765 14557 raft_consensus.cc:2229] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:55.276264 14557 pending_rounds.cc:62] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:55.276427 14557 pending_rounds.cc:68] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744091738165248 op_type: NO_OP noop_request { }
I20240506 19:56:55.276635 14557 raft_consensus.cc:2866] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:55.276808 14557 raft_consensus.cc:2258] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.277508 14557 ts_tablet_manager.cc:1875] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.279012 12781 leader_election.cc:302] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:55.279479 12781 leader_election.cc:341] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 4095f9fc28ff48f68064ec8318ff7b51
I20240506 19:56:55.279639 12781 leader_election.cc:302] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:55.279996 12781 leader_election.cc:341] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: bafd3c4706f74e7da36a8f3538760f43
I20240506 19:56:55.280107 12781 leader_election.cc:302] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.280153 14518 raft_consensus.cc:482] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.280288 14518 raft_consensus.cc:504] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.280443 14561 raft_consensus.cc:2726] T be6e733b6efd4041abf465c7bb788ef5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.280612 14561 raft_consensus.cc:2726] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.281225 14561 raft_consensus.cc:2726] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.281226 14518 leader_election.cc:288] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:55.281404 14518 consensus_queue.cc:1035] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:55.281984 12922 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 997ecba7fbae41178a25bf7af4d152e1
I20240506 19:56:55.282078 12922 leader_election.cc:302] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.282263 14518 raft_consensus.cc:2726] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.262526 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 1006) took 1037 ms. Trace:
I20240506 19:56:55.284104 13246 rpcz_store.cc:276] 0506 19:56:54.224831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.225342 (+ 511us) service_pool.cc:225] Handling call
0506 19:56:55.104472 (+879130us) ts_tablet_manager.cc:1120] Deleting tablet 2ff3435c11854649b7daf74f3d8c33d7
0506 19:56:55.104486 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.122367 (+ 17881us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.209090 (+ 86723us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.262501 (+ 53411us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13317,"tablet-delete.queue_time_us":865768}
I20240506 19:56:55.284667 13246 tablet_replica.cc:330] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.284619 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 2ff3435c11854649b7daf74f3d8c33d7 (table default.Table_6d22301588d54dcd8649bc08a1ade8f6 [id=09c0d79ce0d443079ca27266e1154dbc]) successfully deleted
I20240506 19:56:55.284854 13246 raft_consensus.cc:2229] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.284996 13246 raft_consensus.cc:2258] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.285600 13246 ts_tablet_manager.cc:1875] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.290804 14518 raft_consensus.cc:482] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.291013 14518 raft_consensus.cc:504] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.291970 14518 leader_election.cc:288] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:55.294441 13246 ts_tablet_manager.cc:1888] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.294570 13246 log.cc:1196] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/08dafb661b2443f5afadc6816868bac8
I20240506 19:56:55.296449 13246 ts_tablet_manager.cc:1909] T 08dafb661b2443f5afadc6816868bac8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.296875 13246 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56326 (request call id 1003) took 1072 ms. Trace:
I20240506 19:56:55.296969 13246 rpcz_store.cc:276] 0506 19:56:54.224474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.225515 (+ 1041us) service_pool.cc:225] Handling call
0506 19:56:55.284583 (+1059068us) ts_tablet_manager.cc:1120] Deleting tablet 08dafb661b2443f5afadc6816868bac8
0506 19:56:55.284595 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.288873 (+ 4278us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.294314 (+ 5441us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.296850 (+ 2536us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1058961}
I20240506 19:56:55.297514 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 08dafb661b2443f5afadc6816868bac8 (table default.Table_72cbb579f8c5457ab8ef64876959a3a5 [id=8c16005dc18b44a4918959ca3868f9f0]) successfully deleted
W20240506 19:56:55.297869 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 766d77bb818e429fb9b745e99adea585 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 766d77bb818e429fb9b745e99adea585 already in progress: creating tablet
W20240506 19:56:55.298699 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0fb9c05d703f417780e77359e19e542e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fb9c05d703f417780e77359e19e542e already in progress: creating tablet
W20240506 19:56:55.298941 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 01da81cb9ee840ed9e6640af305eaecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01da81cb9ee840ed9e6640af305eaecc already in progress: creating tablet
W20240506 19:56:55.299196 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 141bca5c816a494b83704a915f0edfd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 141bca5c816a494b83704a915f0edfd1 already in progress: creating tablet
W20240506 19:56:55.299417 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 1980a68925be4d3da1078c78d32da866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1980a68925be4d3da1078c78d32da866 already in progress: creating tablet
W20240506 19:56:55.299749 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2919538141834fb6a83259e29288acdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2919538141834fb6a83259e29288acdd already in progress: creating tablet
W20240506 19:56:55.300025 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 182a3de861984b7d978f15a17275b6b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182a3de861984b7d978f15a17275b6b0 already in progress: creating tablet
I20240506 19:56:55.300094 13246 tablet_replica.cc:330] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.300285 13246 raft_consensus.cc:2229] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.300534 13246 raft_consensus.cc:2258] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.301149 13246 ts_tablet_manager.cc:1875] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.302723 13245 tablet_bootstrap.cc:492] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:55.304170 13245 tablet_bootstrap.cc:654] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.306305 13249 tablet_bootstrap.cc:492] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.306489 13249 ts_tablet_manager.cc:1367] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.001s
I20240506 19:56:55.307423 13249 raft_consensus.cc:348] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.307703 13249 raft_consensus.cc:374] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.307755 13249 raft_consensus.cc:729] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.307981 13249 consensus_queue.cc:260] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:55.308534 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e726aa52406d49d3b5c78f558755ac6e consensus state has no leader
I20240506 19:56:55.315037 13246 ts_tablet_manager.cc:1888] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.315219 13246 log.cc:1196] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:56:55.315554 13246 ts_tablet_manager.cc:1909] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.315874 13249 ts_tablet_manager.cc:1398] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:56:55.315887 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ad11ec97e2384012981d0daae3a8adbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.316242 13249 tablet_bootstrap.cc:492] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:55.316466 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad11ec97e2384012981d0daae3a8adbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.318132 13249 tablet_bootstrap.cc:654] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.318014 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_251bb78261d6402bbf113e3110a57ba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.318557 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_251bb78261d6402bbf113e3110a57ba2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.319994 13244 tablet_bootstrap.cc:492] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.319984 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c4ab6be6ef2d46bdb824d78a7eea1f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.320153 13244 ts_tablet_manager.cc:1367] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.000s
W20240506 19:56:55.320480 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4ab6be6ef2d46bdb824d78a7eea1f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.320981 13246 tablet_replica.cc:330] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.321013 13244 raft_consensus.cc:348] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.321183 13246 raft_consensus.cc:2229] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.321236 13244 raft_consensus.cc:374] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.321363 13244 raft_consensus.cc:729] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.321622 13244 consensus_queue.cc:260] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.322050 13246 raft_consensus.cc:2258] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.322213 13244 ts_tablet_manager.cc:1398] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:55.322490 13244 tablet_bootstrap.cc:492] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.322607 13246 ts_tablet_manager.cc:1875] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.324437 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e84e16df5a724b1e8fe6eb666bfa375a" } new_table_name: "default.Table_d486818ffe5644efa8b0211637048aa3" modify_external_catalogs: true
I20240506 19:56:55.325798 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9ab15c3c5575442181350680a86a5370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.326328 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ab15c3c5575442181350680a86a5370 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:55.329620 12922 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 85bc7cfd006f4d4ca95541dab6873d32
W20240506 19:56:55.329989 12922 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:56:55.330153 12893 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:55.332276 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f8ffb88e0a0c4125b04f4d8d328da3f5 (table default.Table_e737c1fa78974f05a022649a900e2636 [id=e3446e2531334554961b65e81f9bb53c]) successfully deleted
I20240506 19:56:55.332669 14518 consensus_queue.cc:1035] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:55.334221 13246 ts_tablet_manager.cc:1888] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.334630 13246 log.cc:1196] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/bece304e0ba048ecbb2d92aa71fdf707
I20240506 19:56:55.334628 14557 ts_tablet_manager.cc:1888] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:55.334847 14557 log.cc:1196] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:56:55.335107 13246 ts_tablet_manager.cc:1909] T bece304e0ba048ecbb2d92aa71fdf707 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.335706 14557 ts_tablet_manager.cc:1909] T ceeae70e0d1144ea9e3e0f8727c15e6b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.335772 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet bece304e0ba048ecbb2d92aa71fdf707 (table default.Table_4d65e20e3d924618a805ec2be5bb9b29 [id=d78c40589bf3432aa232674727a95dcf]) successfully deleted
I20240506 19:56:55.336153 13246 tablet_replica.cc:330] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
W20240506 19:56:55.336153 12508 catalog_manager.cc:4797] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 99f4861a01404663870ce91978c8af4f because the tablet was not found. No further retry: Not found: Tablet not found: 99f4861a01404663870ce91978c8af4f
I20240506 19:56:55.336380 13246 raft_consensus.cc:2229] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.336587 13246 raft_consensus.cc:2258] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.337282 13246 ts_tablet_manager.cc:1875] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.340268 14557 tablet_replica.cc:330] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.340409 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ceeae70e0d1144ea9e3e0f8727c15e6b (table default.Table_54f3105144e741119c67aec80c408fa5 [id=6369da651bef4fa58ae1c1711da138b8]) successfully deleted
I20240506 19:56:55.345525 13244 tablet_bootstrap.cc:654] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.346482 14557 raft_consensus.cc:2229] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:55.346570 14436 raft_consensus.cc:482] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.346763 14436 raft_consensus.cc:504] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.347321 14557 pending_rounds.cc:62] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:55.347540 14436 leader_election.cc:288] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.347707 13246 ts_tablet_manager.cc:1888] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.347837 13246 log.cc:1196] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/40b9731a473449b19a9e95653a9d0830
I20240506 19:56:55.346297 13242 ts_tablet_manager.cc:1888] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.347997 13242 log.cc:1196] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/fdab4110236e4f5c8690f6d963ff27f7
I20240506 19:56:55.348212 13246 ts_tablet_manager.cc:1909] T 40b9731a473449b19a9e95653a9d0830 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.348349 13242 ts_tablet_manager.cc:1909] T fdab4110236e4f5c8690f6d963ff27f7 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:55.348711 12783 leader_election.cc:341] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f8951bf4873e4382bcd42a52368fb376
I20240506 19:56:55.348825 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 40b9731a473449b19a9e95653a9d0830 (table default.Table_b40bb86f59744dc387a12a83d5dc8b9b [id=c4408f37ca81447585abb9f25a9f4a15]) successfully deleted
I20240506 19:56:55.347697 14557 pending_rounds.cc:68] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744090491965440 op_type: NO_OP noop_request { }
I20240506 19:56:55.354573 13244 tablet_bootstrap.cc:492] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.354902 13244 ts_tablet_manager.cc:1367] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.004s
I20240506 19:56:55.356069 13244 raft_consensus.cc:348] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.356432 13244 raft_consensus.cc:374] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.356580 13244 raft_consensus.cc:729] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.356910 13244 consensus_queue.cc:260] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.357596 13244 ts_tablet_manager.cc:1398] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:55.357753 14557 raft_consensus.cc:2866] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:55.357970 14557 raft_consensus.cc:2258] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.359232 14557 ts_tablet_manager.cc:1875] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.361629 13244 tablet_bootstrap.cc:492] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.361433 12514 catalog_manager.cc:5484] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:55.361935 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:55.362241 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]). Expected version 2 got 0
I20240506 19:56:55.362460 12514 catalog_manager.cc:5484] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:55.362915 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:55.363198 12514 catalog_manager.cc:5484] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:55.363584 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:55.363934 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet dcb357a7eb2147a79cc69d024bd35269 (table default.Table_b35bb26efd3c4ca08971e6c1185a6c0b [id=b6aeaf0e19d746dab4a37005dc753d65]). Expected version 1 got 0
W20240506 19:56:55.363982 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 84011.9us
I20240506 19:56:55.364106 13244 tablet_bootstrap.cc:654] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.364214 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:55.364337 12781 leader_election.cc:302] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.364606 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
W20240506 19:56:55.364784 12781 leader_election.cc:341] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 03df4632adfa4b0ab999aeaa5bb4e1f4
I20240506 19:56:55.364910 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:55.364921 12781 leader_election.cc:302] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.365103 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:55.365237 14436 raft_consensus.cc:2726] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.365475 14436 raft_consensus.cc:2726] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.367511 12824 tablet_service.cc:1467] Processing CreateTablet for tablet ff722c670f654025b25ad483d22d8d25 (DEFAULT_TABLE table=default.Table_62d73cc2b6a847ebb8f40ad1ce877813 [id=c2199b4285764218b8785b6dd4f4711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.367734 12952 tablet_service.cc:1467] Processing CreateTablet for tablet ff722c670f654025b25ad483d22d8d25 (DEFAULT_TABLE table=default.Table_62d73cc2b6a847ebb8f40ad1ce877813 [id=c2199b4285764218b8785b6dd4f4711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.367949 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff722c670f654025b25ad483d22d8d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.368144 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff722c670f654025b25ad483d22d8d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.365265 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:55.368930 12514 catalog_manager.cc:5484] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:55.369346 12514 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]). Expected version 1 got 0
I20240506 19:56:55.373109 13090 tablet_service.cc:1467] Processing CreateTablet for tablet ff722c670f654025b25ad483d22d8d25 (DEFAULT_TABLE table=default.Table_62d73cc2b6a847ebb8f40ad1ce877813 [id=c2199b4285764218b8785b6dd4f4711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.373569 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff722c670f654025b25ad483d22d8d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.376860 12965 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad14a17520a49beb3682b8fd04f9e79 (DEFAULT_TABLE table=default.Table_cae804c3fdd14f67bde6f49878f4d248 [id=ebcb68f58ca747c7afb08c4c4a900685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.377393 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad14a17520a49beb3682b8fd04f9e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.378123 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad14a17520a49beb3682b8fd04f9e79 (DEFAULT_TABLE table=default.Table_cae804c3fdd14f67bde6f49878f4d248 [id=ebcb68f58ca747c7afb08c4c4a900685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.378477 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad14a17520a49beb3682b8fd04f9e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.380173 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad14a17520a49beb3682b8fd04f9e79 (DEFAULT_TABLE table=default.Table_cae804c3fdd14f67bde6f49878f4d248 [id=ebcb68f58ca747c7afb08c4c4a900685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.380656 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad14a17520a49beb3682b8fd04f9e79. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:55.381315 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 418e02201bfc4a34a22612c2edc3e05f consensus state has no leader
I20240506 19:56:55.381551 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef (DEFAULT_TABLE table=default.Table_ab095fce093c440cb77fc15d75f7af97 [id=5a2a63588e864676bbbaf4ab586a819d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.381860 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.382104 12518 master_service.cc:946] ReplaceTablet: received request to replace tablet 607b3fc2c17b4793af9b84c19066ec5e from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:55.383045 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 904f11fe9eac4fcea9832218a4bed98b from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:55.385591 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef (DEFAULT_TABLE table=default.Table_ab095fce093c440cb77fc15d75f7af97 [id=5a2a63588e864676bbbaf4ab586a819d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.385986 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.396483 13097 tablet_service.cc:1467] Processing CreateTablet for tablet b77db13623e34ef0aeee0b4a19eaebb2 (DEFAULT_TABLE table=default.Table_76b20d55cba5431aa5b3b38c9eca1e56 [id=860954ec42cf41e68fad33cb0b6a1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.396545 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef (DEFAULT_TABLE table=default.Table_ab095fce093c440cb77fc15d75f7af97 [id=5a2a63588e864676bbbaf4ab586a819d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.396951 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77db13623e34ef0aeee0b4a19eaebb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.397063 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.398681 14557 ts_tablet_manager.cc:1888] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:55.398808 14557 log.cc:1196] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/796beb8e0a2c41d698bd4c05796263f1
I20240506 19:56:55.399169 14557 ts_tablet_manager.cc:1909] T 796beb8e0a2c41d698bd4c05796263f1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.400969 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fdebb50309c74888b6a893ac37f9cb9c" } new_table_name: "default.Table_fd50aaa5e06941b0b3fbb1d2e4966cbb" modify_external_catalogs: true
I20240506 19:56:55.401435 13101 tablet_service.cc:1467] Processing CreateTablet for tablet e77180b913ad4d2db611a925d4f2208f (DEFAULT_TABLE table=default.Table_550853d348524a21be958c7e260c141b [id=64fa4dbbafd6488bbca91ed2c037a762]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.401824 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77180b913ad4d2db611a925d4f2208f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.401795 12956 tablet_service.cc:1467] Processing CreateTablet for tablet b77db13623e34ef0aeee0b4a19eaebb2 (DEFAULT_TABLE table=default.Table_76b20d55cba5431aa5b3b38c9eca1e56 [id=860954ec42cf41e68fad33cb0b6a1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.402181 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77db13623e34ef0aeee0b4a19eaebb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.402498 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a6c3a4c9339a480db1e49222fe7e0368" } new_table_name: "default.Table_16db473ae5ed4bd0ab2c8795b490f5cf" modify_external_catalogs: true
I20240506 19:56:55.402993 12816 tablet_service.cc:1467] Processing CreateTablet for tablet e77180b913ad4d2db611a925d4f2208f (DEFAULT_TABLE table=default.Table_550853d348524a21be958c7e260c141b [id=64fa4dbbafd6488bbca91ed2c037a762]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.403635 12817 tablet_service.cc:1467] Processing CreateTablet for tablet b77db13623e34ef0aeee0b4a19eaebb2 (DEFAULT_TABLE table=default.Table_76b20d55cba5431aa5b3b38c9eca1e56 [id=860954ec42cf41e68fad33cb0b6a1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.404098 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ea32a023a6aa4ac1852584d211ede823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.404639 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea32a023a6aa4ac1852584d211ede823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.418061 14557 tablet_replica.cc:330] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.418200 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 796beb8e0a2c41d698bd4c05796263f1 (table default.Table_58c604462fcb4b9cab62687d64877b02 [id=923944e6f77349c3b09903ba96a887fc]) successfully deleted
I20240506 19:56:55.418319 14557 raft_consensus.cc:2229] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:55.418687 14557 pending_rounds.cc:62] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:55.418772 14557 pending_rounds.cc:68] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744090800418816 op_type: NO_OP noop_request { }
I20240506 19:56:55.418911 14557 raft_consensus.cc:2866] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:55.418887 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020" } modify_external_catalogs: true
I20240506 19:56:55.419023 14557 raft_consensus.cc:2258] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.419157 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a5cf8ba09d8b4c4b93f45fa8e3643020" } modify_external_catalogs: true
I20240506 19:56:55.419617 14557 ts_tablet_manager.cc:1875] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.422065 13245 tablet_bootstrap.cc:492] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:55.422222 13245 ts_tablet_manager.cc:1367] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.120s user 0.003s sys 0.000s
I20240506 19:56:55.422951 13245 raft_consensus.cc:348] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.423149 13245 raft_consensus.cc:374] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.423221 13245 raft_consensus.cc:729] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.423449 13245 consensus_queue.cc:260] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.424001 13245 ts_tablet_manager.cc:1398] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:55.424273 13245 tablet_bootstrap.cc:492] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:55.425616 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77180b913ad4d2db611a925d4f2208f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.425902 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77db13623e34ef0aeee0b4a19eaebb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.435642 14518 raft_consensus.cc:482] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.435873 14518 raft_consensus.cc:504] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.437131 14518 leader_election.cc:288] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:55.437999 13085 tablet_service.cc:1467] Processing CreateTablet for tablet b0086223f09b403b9531bfad4bb15f47 (DEFAULT_TABLE table=default.Table_d61d010cfd0a45fc9c7413c61e051f1a [id=d4ec4f62a41b458faa3e28b41a2523cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.438365 12956 tablet_service.cc:1467] Processing CreateTablet for tablet b0086223f09b403b9531bfad4bb15f47 (DEFAULT_TABLE table=default.Table_d61d010cfd0a45fc9c7413c61e051f1a [id=d4ec4f62a41b458faa3e28b41a2523cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.438700 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0086223f09b403b9531bfad4bb15f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.438786 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0086223f09b403b9531bfad4bb15f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.438746 12817 tablet_service.cc:1467] Processing CreateTablet for tablet b0086223f09b403b9531bfad4bb15f47 (DEFAULT_TABLE table=default.Table_d61d010cfd0a45fc9c7413c61e051f1a [id=d4ec4f62a41b458faa3e28b41a2523cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.439154 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0086223f09b403b9531bfad4bb15f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.439235 12962 tablet_service.cc:1467] Processing CreateTablet for tablet 68027db821bf4ad3beffc7d160c98988 (DEFAULT_TABLE table=default.Table_54845e03ef4e4cc7b472aeee0c2b4c42 [id=2e805268d6aa44a681a952f654d3055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.439599 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68027db821bf4ad3beffc7d160c98988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.442389 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 7f460e0fc06a4972a2379b08fdedfc10 (DEFAULT_TABLE table=default.Table_825048dc8d55425b9dc8c3787acd53ab [id=78308bf582904922bfa15bec097a31a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.443048 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f460e0fc06a4972a2379b08fdedfc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.445624 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 9be0f4fdf226446e974dc139fd05babb (DEFAULT_TABLE table=default.Table_24269238bd23445a93f9bf17765610af [id=a02df3672ee24494842ca04c4d67e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.446031 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be0f4fdf226446e974dc139fd05babb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.446498 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 7f460e0fc06a4972a2379b08fdedfc10 (DEFAULT_TABLE table=default.Table_825048dc8d55425b9dc8c3787acd53ab [id=78308bf582904922bfa15bec097a31a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.446863 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f460e0fc06a4972a2379b08fdedfc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.448979 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 68027db821bf4ad3beffc7d160c98988 (DEFAULT_TABLE table=default.Table_54845e03ef4e4cc7b472aeee0c2b4c42 [id=2e805268d6aa44a681a952f654d3055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.449923 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68027db821bf4ad3beffc7d160c98988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.450698 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 7f460e0fc06a4972a2379b08fdedfc10 (DEFAULT_TABLE table=default.Table_825048dc8d55425b9dc8c3787acd53ab [id=78308bf582904922bfa15bec097a31a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.451236 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f460e0fc06a4972a2379b08fdedfc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.453274 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 68027db821bf4ad3beffc7d160c98988 (DEFAULT_TABLE table=default.Table_54845e03ef4e4cc7b472aeee0c2b4c42 [id=2e805268d6aa44a681a952f654d3055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.453655 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68027db821bf4ad3beffc7d160c98988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.453927 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 9be0f4fdf226446e974dc139fd05babb (DEFAULT_TABLE table=default.Table_24269238bd23445a93f9bf17765610af [id=a02df3672ee24494842ca04c4d67e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.454293 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be0f4fdf226446e974dc139fd05babb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.463905 13089 tablet_service.cc:1467] Processing CreateTablet for tablet f637032aefd1473b907bd457cf5fd104 (DEFAULT_TABLE table=default.Table_fb5492528b144beeaec025a770bc00f0 [id=672c95de014c4388a14a6605c25de148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.464803 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f637032aefd1473b907bd457cf5fd104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.466225 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 9be0f4fdf226446e974dc139fd05babb (DEFAULT_TABLE table=default.Table_24269238bd23445a93f9bf17765610af [id=a02df3672ee24494842ca04c4d67e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.466327 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8b7c300d574b539169209f84820326 (DEFAULT_TABLE table=default.Table_c2d2254e8afc4f8a80b105ceffab0b47 [id=3614ed5dd41b467aa26de18b1b04ed08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.466789 12950 tablet_service.cc:1467] Processing CreateTablet for tablet f637032aefd1473b907bd457cf5fd104 (DEFAULT_TABLE table=default.Table_fb5492528b144beeaec025a770bc00f0 [id=672c95de014c4388a14a6605c25de148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.467496 12953 tablet_service.cc:1467] Processing CreateTablet for tablet c1d8fbd6adc24563a0c176184177e849 (DEFAULT_TABLE table=default.Table_edb2261b3c4944cb8218f60533811bdb [id=64b49549c03648dab3e6b25e726405bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.468132 12958 tablet_service.cc:1467] Processing CreateTablet for tablet cec4f6112faa4f34afde9449636216b4 (DEFAULT_TABLE table=default.Table_10189ed452404f5d90cc93ca6b59e7e9 [id=ca65b8a1c8ef4c9b8a04d85fee307005]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.468690 14436 raft_consensus.cc:482] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.468915 14436 raft_consensus.cc:504] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.464280 13087 tablet_service.cc:1467] Processing CreateTablet for tablet c1d8fbd6adc24563a0c176184177e849 (DEFAULT_TABLE table=default.Table_edb2261b3c4944cb8218f60533811bdb [id=64b49549c03648dab3e6b25e726405bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.469667 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d8fbd6adc24563a0c176184177e849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.469767 14436 leader_election.cc:288] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.470584 12812 tablet_service.cc:1467] Processing CreateTablet for tablet f637032aefd1473b907bd457cf5fd104 (DEFAULT_TABLE table=default.Table_fb5492528b144beeaec025a770bc00f0 [id=672c95de014c4388a14a6605c25de148]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.470872 13245 tablet_bootstrap.cc:654] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.471024 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f637032aefd1473b907bd457cf5fd104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.473953 12825 tablet_service.cc:1467] Processing CreateTablet for tablet c1d8fbd6adc24563a0c176184177e849 (DEFAULT_TABLE table=default.Table_edb2261b3c4944cb8218f60533811bdb [id=64b49549c03648dab3e6b25e726405bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.473975 12826 tablet_service.cc:1467] Processing CreateTablet for tablet cec4f6112faa4f34afde9449636216b4 (DEFAULT_TABLE table=default.Table_10189ed452404f5d90cc93ca6b59e7e9 [id=ca65b8a1c8ef4c9b8a04d85fee307005]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.474399 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d8fbd6adc24563a0c176184177e849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.474637 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec4f6112faa4f34afde9449636216b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.477813 12815 tablet_service.cc:1467] Processing CreateTablet for tablet da6c6a32aa044a3ba0ae1e79b10d8bd9 (DEFAULT_TABLE table=default.Table_d56b1e730d4c4eee9b775d01f57057c5 [id=c96795bdf3654876bd8d7ea4306f7b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.478227 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6c6a32aa044a3ba0ae1e79b10d8bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.481626 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8b7c300d574b539169209f84820326 (DEFAULT_TABLE table=default.Table_c2d2254e8afc4f8a80b105ceffab0b47 [id=3614ed5dd41b467aa26de18b1b04ed08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.482138 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8b7c300d574b539169209f84820326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.482527 13099 tablet_service.cc:1467] Processing CreateTablet for tablet cec4f6112faa4f34afde9449636216b4 (DEFAULT_TABLE table=default.Table_10189ed452404f5d90cc93ca6b59e7e9 [id=ca65b8a1c8ef4c9b8a04d85fee307005]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.482909 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec4f6112faa4f34afde9449636216b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:55.483484 12783 leader_election.cc:341] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b899fb38bfed4e02b3708f6abec80e0a
I20240506 19:56:55.484555 14557 ts_tablet_manager.cc:1888] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:55.484666 14557 log.cc:1196] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a9e90b93023e40b3bafbddf7bbcc10b2
I20240506 19:56:55.485004 14557 ts_tablet_manager.cc:1909] T a9e90b93023e40b3bafbddf7bbcc10b2 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.487067 13096 tablet_service.cc:1467] Processing CreateTablet for tablet da6c6a32aa044a3ba0ae1e79b10d8bd9 (DEFAULT_TABLE table=default.Table_d56b1e730d4c4eee9b775d01f57057c5 [id=c96795bdf3654876bd8d7ea4306f7b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.487443 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6c6a32aa044a3ba0ae1e79b10d8bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.489039 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8b7c300d574b539169209f84820326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.489550 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec4f6112faa4f34afde9449636216b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.490005 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d8fbd6adc24563a0c176184177e849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.490389 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be0f4fdf226446e974dc139fd05babb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.490805 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f637032aefd1473b907bd457cf5fd104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.491092 14557 tablet_replica.cc:330] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.491313 14557 raft_consensus.cc:2229] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:55.491663 14557 pending_rounds.cc:62] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:55.491719 14557 pending_rounds.cc:68] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744094072520704 op_type: NO_OP noop_request { }
I20240506 19:56:55.491822 14557 raft_consensus.cc:2866] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:55.491892 14557 raft_consensus.cc:2258] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.492451 14557 ts_tablet_manager.cc:1875] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.463156 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8b7c300d574b539169209f84820326 (DEFAULT_TABLE table=default.Table_c2d2254e8afc4f8a80b105ceffab0b47 [id=3614ed5dd41b467aa26de18b1b04ed08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.494431 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8b7c300d574b539169209f84820326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.500101 13096 tablet_service.cc:1467] Processing CreateTablet for tablet fb2b44e607364e9bb86ad214e1911e47 (DEFAULT_TABLE table=default.Table_ef58202ae07041c18e495f8a92ec3eb5 [id=0efca22312684fd0ba5a7db69f4fed15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.500495 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2b44e607364e9bb86ad214e1911e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.525476 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a9e90b93023e40b3bafbddf7bbcc10b2 (table default.Table_6b562516d9b845789bf9d901c015e37c [id=11d0ad6e93ef421182aaac061edb8501]) successfully deleted
I20240506 19:56:55.526775 12805 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50488 (request call id 164) took 1014 ms. Trace:
I20240506 19:56:55.526892 12805 rpcz_store.cc:276] 0506 19:56:54.512785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.513668 (+ 883us) service_pool.cc:225] Handling call
0506 19:56:55.526747 (+1013079us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 19:56:55.533659 12817 tablet_service.cc:1467] Processing CreateTablet for tablet fb2b44e607364e9bb86ad214e1911e47 (DEFAULT_TABLE table=default.Table_ef58202ae07041c18e495f8a92ec3eb5 [id=0efca22312684fd0ba5a7db69f4fed15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.535584 12956 tablet_service.cc:1467] Processing CreateTablet for tablet da6c6a32aa044a3ba0ae1e79b10d8bd9 (DEFAULT_TABLE table=default.Table_d56b1e730d4c4eee9b775d01f57057c5 [id=c96795bdf3654876bd8d7ea4306f7b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.536101 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6c6a32aa044a3ba0ae1e79b10d8bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.538102 14436 raft_consensus.cc:482] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.538254 14436 raft_consensus.cc:504] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.538926 14436 leader_election.cc:288] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:55.540098 12783 leader_election.cc:341] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b3e50bde3564493eb7267acb42cb8592
I20240506 19:56:55.553171 12950 tablet_service.cc:1467] Processing CreateTablet for tablet fb2b44e607364e9bb86ad214e1911e47 (DEFAULT_TABLE table=default.Table_ef58202ae07041c18e495f8a92ec3eb5 [id=0efca22312684fd0ba5a7db69f4fed15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.538380 14557 ts_tablet_manager.cc:1888] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:55.553938 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2b44e607364e9bb86ad214e1911e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.557374 12952 tablet_service.cc:1467] Processing CreateTablet for tablet e77180b913ad4d2db611a925d4f2208f (DEFAULT_TABLE table=default.Table_550853d348524a21be958c7e260c141b [id=64fa4dbbafd6488bbca91ed2c037a762]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.557868 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77180b913ad4d2db611a925d4f2208f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.559226 14557 log.cc:1196] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/04738def56324a368f8182543107e32e
I20240506 19:56:55.560662 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 0ec1ac39f62b4e15a267f5db2a6166c6
I20240506 19:56:55.561249 12519 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 1235) took 1025 ms. Trace:
I20240506 19:56:55.561394 12519 rpcz_store.cc:276] 0506 19:56:54.536065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.216593 (+680528us) service_pool.cc:225] Handling call
0506 19:56:55.242834 (+ 26241us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x616000021d54. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 00007f8feed4915d 00007f8feed49253 00007f8feba7ad41 00007f8ff366734d 00007f8ff37f4ca6 00007f8fea7e6f66 00007f8fea7e6c95 00007f8fe6f1104e 00007f8fe6f0ff2e 00007f8fe6f16405 00007f8fe6f16d2f
0506 19:56:55.243218 (+ 384us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:55.243404 (+ 186us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:55.243466 (+ 62us) catalog_manager.cc:2758] Locking tablets
0506 19:56:55.354689 (+111223us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:55.560431 (+205742us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:55.560536 (+ 105us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:55.560585 (+ 49us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:55.561198 (+ 613us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:55.355410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:55.355540 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:55.355546 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:55.355548 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:55.355665 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:55.355677 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:55.355682 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:55.355781 (+ 99us) tablet.cc:699] Row locks acquired
0506 19:56:55.355783 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:55.355819 (+ 36us) write_op.cc:270] Start()
0506 19:56:55.355849 (+ 30us) write_op.cc:276] Timestamp: P: 1715025415355953 usec, L: 0
0506 19:56:55.355853 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:55.356017 (+ 164us) log.cc:842] Serialized 762 byte log entry
0506 19:56:55.559124 (+203107us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.559268 (+ 144us) write_op.cc:301] APPLY: starting
0506 19:56:55.559291 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.559355 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.559377 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.559501 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.559534 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:55.559543 (+ 9us) write_op.cc:312] APPLY: finished
0506 19:56:55.559653 (+ 110us) log.cc:842] Serialized 29 byte log entry
0506 19:56:55.559739 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.559763 (+ 24us) write_op.cc:454] Released schema lock
0506 19:56:55.559773 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.559842 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.559846 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":59730304,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":640,"apply.run_wall_time_us":639,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":677,"prepare.run_wall_time_us":677,"replication_time_us":203243,"spinlock_wait_cycles":3328}]]}
I20240506 19:56:55.562265 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9f9b64059bcb4cb0b125b84b6f163ef2" } modify_external_catalogs: true
I20240506 19:56:55.562543 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9f9b64059bcb4cb0b125b84b6f163ef2" } modify_external_catalogs: true
I20240506 19:56:55.562855 14557 ts_tablet_manager.cc:1909] T 04738def56324a368f8182543107e32e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.563936 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 04738def56324a368f8182543107e32e (table default.Table_0c42cb0cd7e349fa86150517492387bb [id=52a0a57c5c19479e90684d073873755b]) successfully deleted
I20240506 19:56:55.356475 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 908) took 2847 ms. Trace:
I20240506 19:56:55.569386 13242 rpcz_store.cc:276] 0506 19:56:52.508526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.509785 (+ 1259us) service_pool.cc:225] Handling call
0506 19:56:55.269285 (+2759500us) ts_tablet_manager.cc:1120] Deleting tablet fdab4110236e4f5c8690f6d963ff27f7
0506 19:56:55.269316 (+ 31us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.276346 (+ 7030us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.327334 (+ 50988us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.356412 (+ 29078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2759260}
I20240506 19:56:55.569687 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:55.569887 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 909) took 3030 ms. Trace:
I20240506 19:56:55.569969 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:55.569967 13242 rpcz_store.cc:276] 0506 19:56:52.539237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.539431 (+ 194us) service_pool.cc:225] Handling call
0506 19:56:55.569731 (+3030300us) ts_tablet_manager.cc:1120] Deleting tablet 4f4213fa718b491881f304b4e8f0bf7a
0506 19:56:55.569743 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.569866 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030104}
I20240506 19:56:55.570151 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 838d9e7986164ffd916d2d842dc61f6e (table default.Table_b7e1ec0e7d9645cc9cf705d1ed10fd5d [id=f424bbb43b2a47449f998c21dae05cb9]). Expected version 2 got 0
I20240506 19:56:55.570225 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 910) took 3030 ms. Trace:
I20240506 19:56:55.570298 13242 rpcz_store.cc:276] 0506 19:56:52.539364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.539651 (+ 287us) service_pool.cc:225] Handling call
0506 19:56:55.570144 (+3030493us) ts_tablet_manager.cc:1120] Deleting tablet f424e713b96d43cdac6597c20bd5fc73
0506 19:56:55.570152 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.570213 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030400}
I20240506 19:56:55.570355 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet dcb357a7eb2147a79cc69d024bd35269 (table default.Table_b35bb26efd3c4ca08971e6c1185a6c0b [id=b6aeaf0e19d746dab4a37005dc753d65]). Expected version 1 got 0
I20240506 19:56:55.570555 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:55.570557 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 911) took 2889 ms. Trace:
I20240506 19:56:55.570618 13242 rpcz_store.cc:276] 0506 19:56:52.681351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.681405 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:55.570469 (+2889064us) ts_tablet_manager.cc:1120] Deleting tablet e6b24176d9d941d4a9dfb97e24b91ab6
0506 19:56:55.570478 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.570544 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2888899}
I20240506 19:56:55.570828 13242 tablet_replica.cc:330] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.571000 13242 raft_consensus.cc:2229] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.570995 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]). Expected version 2 got 0
I20240506 19:56:55.571141 13242 raft_consensus.cc:2258] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.571681 13242 ts_tablet_manager.cc:1875] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.573412 12513 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]). Expected version 2 got 0
I20240506 19:56:55.575055 12516 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59] alter complete (version 1)
W20240506 19:56:55.577152 12516 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.193:41395 (request call id 171) took 1352 ms (client timeout 1000 ms). Trace:
W20240506 19:56:55.577286 12516 rpcz_store.cc:269] 0506 19:56:54.224220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.248402 (+ 24182us) service_pool.cc:225] Handling call
0506 19:56:55.577107 (+1328705us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:54.293023 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:54.293148 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:54.293157 (+ 9us) write_op.cc:435] Acquired schema lock
0506 19:56:54.293161 (+ 4us) tablet.cc:660] Decoding operations
0506 19:56:54.293283 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:54.293319 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:54.293327 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:54.293407 (+ 80us) tablet.cc:699] Row locks acquired
0506 19:56:54.293411 (+ 4us) write_op.cc:260] PREPARE: finished
0506 19:56:54.293459 (+ 48us) write_op.cc:270] Start()
0506 19:56:54.293493 (+ 34us) write_op.cc:276] Timestamp: P: 1715025414293571 usec, L: 0
0506 19:56:54.293496 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:54.293679 (+ 183us) log.cc:842] Serialized 356 byte log entry
0506 19:56:54.322098 (+ 28419us) op_driver.cc:444] REPLICATION: finished
0506 19:56:54.323052 (+ 954us) write_op.cc:301] APPLY: starting
0506 19:56:54.323064 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:54.323102 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:54.323127 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:54.323164 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:54.323187 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:54.323194 (+ 7us) write_op.cc:312] APPLY: finished
0506 19:56:54.323238 (+ 44us) log.cc:842] Serialized 23 byte log entry
0506 19:56:54.323280 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:54.323291 (+ 11us) write_op.cc:454] Released schema lock
0506 19:56:54.323296 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:54.323333 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:54.323337 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 19:56:54.643659 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:54.643773 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:54.643779 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:54.643782 (+ 3us) tablet.cc:660] Decoding operations
0506 19:56:54.643894 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:54.643906 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:54.643911 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:54.643987 (+ 76us) tablet.cc:699] Row locks acquired
0506 19:56:54.643990 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:54.644029 (+ 39us) write_op.cc:270] Start()
0506 19:56:54.644062 (+ 33us) write_op.cc:276] Timestamp: P: 1715025414644083 usec, L: 0
0506 19:56:54.644068 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 19:56:54.644248 (+ 180us) log.cc:842] Serialized 356 byte log entry
0506 19:56:55.181670 (+537422us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.181761 (+ 91us) write_op.cc:301] APPLY: starting
0506 19:56:55.181779 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.181827 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.181857 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.181925 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.181968 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:55.181979 (+ 11us) write_op.cc:312] APPLY: finished
0506 19:56:55.182066 (+ 87us) log.cc:842] Serialized 23 byte log entry
0506 19:56:55.182144 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.182672 (+ 528us) write_op.cc:454] Released schema lock
0506 19:56:55.182681 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.182826 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.182830 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 19:56:55.192314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:55.192463 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:55.192472 (+ 9us) write_op.cc:435] Acquired schema lock
0506 19:56:55.192477 (+ 5us) tablet.cc:660] Decoding operations
0506 19:56:55.192593 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:55.192605 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:55.192611 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:55.192693 (+ 82us) tablet.cc:699] Row locks acquired
0506 19:56:55.192696 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:55.192729 (+ 33us) write_op.cc:270] Start()
0506 19:56:55.192761 (+ 32us) write_op.cc:276] Timestamp: P: 1715025415192799 usec, L: 0
0506 19:56:55.192765 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:55.220251 (+ 27486us) log.cc:842] Serialized 356 byte log entry
0506 19:56:55.220478 (+ 227us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x616000021d54. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 000000000095519a 00007f8feba4e162 00007f8fecdb6e32 00007f8fecdb5959 00007f8fecdbd86b 00007f8fecdbd6e8 0000000000a60bcc 00007f8fe024fb74 00007f8fe025319b 00007f8fe0253018 0000000000a60bcc
0506 19:56:55.261574 (+ 41096us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.262509 (+ 935us) write_op.cc:301] APPLY: starting
0506 19:56:55.262542 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.262591 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.262593 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.262673 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.262724 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:55.262735 (+ 11us) write_op.cc:312] APPLY: finished
0506 19:56:55.262907 (+ 172us) log.cc:842] Serialized 23 byte log entry
0506 19:56:55.263184 (+ 277us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.263207 (+ 23us) write_op.cc:454] Released schema lock
0506 19:56:55.263215 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.263343 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.263347 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1451,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":915,"apply.run_cpu_time_us":324,"apply.run_wall_time_us":324,"mutex_wait_us":41787,"num_ops":1,"prepare.queue_time_us":41874,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":735,"replication_time_us":28582}],["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":529,"apply.run_wall_time_us":1114,"num_ops":1,"prepare.queue_time_us":262,"prepare.run_cpu_time_us":674,"prepare.run_wall_time_us":673,"replication_time_us":537579}],["op",{"apply.queue_time_us":864,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":903,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":1286,"prepare.run_wall_time_us":28223,"raft.queue_time_us":75,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":235,"replication_time_us":68780,"spinlock_wait_cycles":62709632}]]}
I20240506 19:56:55.578797 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e33d6f2f5f404ec6be38fd97576e41ad" } modify_external_catalogs: true
I20240506 19:56:55.579012 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e33d6f2f5f404ec6be38fd97576e41ad" } modify_external_catalogs: true
I20240506 19:56:55.585595 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2b44e607364e9bb86ad214e1911e47. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:55.587167 12781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 222462us
I20240506 19:56:55.587498 14518 raft_consensus.cc:482] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.587730 14518 raft_consensus.cc:504] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.588810 14518 leader_election.cc:288] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:55.598637 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet fdab4110236e4f5c8690f6d963ff27f7 (table default.Table_b4ce4fd613b049df9285a95fe23671c1 [id=89681fb9b58c4f429a4f94873f3827ff]) successfully deleted
W20240506 19:56:55.598731 12781 leader_election.cc:341] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 17fb091762144f6591152680a6acd29e
W20240506 19:56:55.599028 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4f4213fa718b491881f304b4e8f0bf7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f4213fa718b491881f304b4e8f0bf7a already in progress: creating tablet
W20240506 19:56:55.599320 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f424e713b96d43cdac6597c20bd5fc73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f424e713b96d43cdac6597c20bd5fc73 already in progress: creating tablet
W20240506 19:56:55.599655 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet e6b24176d9d941d4a9dfb97e24b91ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6b24176d9d941d4a9dfb97e24b91ab6 already in progress: creating tablet
I20240506 19:56:55.600217 12781 leader_election.cc:302] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.602430 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.602700 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de7397db1e74b4488a3f9a70136bccb with delete_type TABLET_DATA_DELETED (replaced by tablet 65315d8ac5ff408bae8593681335e5b1) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.603399 14436 raft_consensus.cc:2726] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.603468 14561 raft_consensus.cc:482] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.603559 14436 raft_consensus.cc:482] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.603650 14436 raft_consensus.cc:504] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.603631 14561 raft_consensus.cc:504] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.604287 14436 leader_election.cc:288] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.604363 14561 leader_election.cc:288] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.605191 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bc7cfd006f4d4ca95541dab6873d32" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:55.605862 12783 leader_election.cc:341] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 63f56d2180b1472a8168fb3a63589065
I20240506 19:56:55.605940 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26e6b8115644cabe6532c7be1bbe95" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:55.606566 14518 raft_consensus.cc:482] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.606752 14518 raft_consensus.cc:504] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:55.606977 12847 raft_consensus.cc:3037] T b35231d3f1bb4348aa7af3c71b809cd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:55.607658 14518 leader_election.cc:288] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:55.619418 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:55.620168 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ec1ac39f62b4e15a267f5db2a6166c6 already in progress: creating tablet
W20240506 19:56:55.620150 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 76561.2us
W20240506 19:56:55.622438 12783 leader_election.cc:341] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 9bb90bf767c94d73ace6411cfd31d65c
W20240506 19:56:55.626428 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7525faf76e174113aa00d5b53aa6ea22 consensus state has no leader
I20240506 19:56:55.626837 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f44e310a873e4dd28f57ab5ff4e28050" } modify_external_catalogs: true
I20240506 19:56:55.627022 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f44e310a873e4dd28f57ab5ff4e28050" } modify_external_catalogs: true
W20240506 19:56:55.628405 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a4a9693c9fb411db07847fede883920 consensus state has no leader
I20240506 19:56:55.628756 12522 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_373183be2430417d8304f5e10cc72a57" } new_table_name: "default.Table_223163dff40648e9824d80b4ca3a8894" modify_external_catalogs: true
I20240506 19:56:55.630236 12520 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6b08752503f442de813189d8b7dea7b9" } new_table_name: "default.Table_3be44884b95d45b4ae8935641eb54286" modify_external_catalogs: true
I20240506 19:56:55.631294 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 904f11fe9eac4fcea9832218a4bed98b
I20240506 19:56:55.631752 12515 catalog_manager.cc:6554] ReplaceTablet: tablet 904f11fe9eac4fcea9832218a4bed98b deleted and replaced by tablet 858d58e1b2d64f47a097bb5eb7a236c7
I20240506 19:56:55.632081 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0f8c273dcd61423a9055682950543663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.632615 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f8c273dcd61423a9055682950543663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.633760 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet b743d0c18c9149fca7da961dade6ad80
I20240506 19:56:55.634459 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_21bad439d300445a8251e13b351a3b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.634920 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21bad439d300445a8251e13b351a3b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:55.626703 12922 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 08dafb661b2443f5afadc6816868bac8
I20240506 19:56:55.638430 13245 tablet_bootstrap.cc:492] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:55.638592 13245 ts_tablet_manager.cc:1367] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.214s user 0.013s sys 0.000s
I20240506 19:56:55.639163 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet b743d0c18c9149fca7da961dade6ad80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.639397 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 904f11fe9eac4fcea9832218a4bed98b with delete_type TABLET_DATA_DELETED (replaced by tablet 858d58e1b2d64f47a097bb5eb7a236c7) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.639380 13245 raft_consensus.cc:348] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.639914 13245 raft_consensus.cc:374] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.640008 13245 raft_consensus.cc:729] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.640288 13245 consensus_queue.cc:260] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.640887 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet b743d0c18c9149fca7da961dade6ad80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.642752 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 904f11fe9eac4fcea9832218a4bed98b with delete_type TABLET_DATA_DELETED (replaced by tablet 858d58e1b2d64f47a097bb5eb7a236c7) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.643013 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet b743d0c18c9149fca7da961dade6ad80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.646134 13245 ts_tablet_manager.cc:1398] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:55.646569 13245 tablet_bootstrap.cc:492] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:55.648092 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 904f11fe9eac4fcea9832218a4bed98b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 904f11fe9eac4fcea9832218a4bed98b already in progress: creating tablet
W20240506 19:56:55.648475 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet b743d0c18c9149fca7da961dade6ad80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b743d0c18c9149fca7da961dade6ad80 already in progress: creating tablet
I20240506 19:56:55.649737 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 904f11fe9eac4fcea9832218a4bed98b with delete_type TABLET_DATA_DELETED (replaced by tablet 858d58e1b2d64f47a097bb5eb7a236c7) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.650277 13245 tablet_bootstrap.cc:654] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.651609 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d] alter complete (version 1)
I20240506 19:56:55.653244 12513 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_1f4f4333e9c147919940536b669bffd3 [id=873dfe33ce7141f2bc25d7575a63fc9f] alter complete (version 2)
I20240506 19:56:55.653929 12514 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11] alter complete (version 2)
I20240506 19:56:55.655473 14398 raft_consensus.cc:482] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.655599 14398 raft_consensus.cc:504] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.655951 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e32a40abc4e74586af79474e4b1675c5
I20240506 19:56:55.656371 14398 leader_election.cc:288] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.656791 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_728d3e1c6e2f499fbbd7bc0994b341a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.657213 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20aa06290b9a4814a006c423ccd8fc46" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:55.657404 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_728d3e1c6e2f499fbbd7bc0994b341a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:55.657902 13058 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.660647 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5e0d2620e9b044d4a5393905dd527e11
I20240506 19:56:55.661479 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2094814385574142ae32d1a198f4b70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.662026 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2094814385574142ae32d1a198f4b70b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.662323 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet e32a40abc4e74586af79474e4b1675c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.662557 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0d2620e9b044d4a5393905dd527e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:55.663373 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet e32a40abc4e74586af79474e4b1675c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e32a40abc4e74586af79474e4b1675c5 already in progress: creating tablet
I20240506 19:56:55.663641 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0d2620e9b044d4a5393905dd527e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:55.663828 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5e0d2620e9b044d4a5393905dd527e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e0d2620e9b044d4a5393905dd527e11 already in progress: creating tablet
I20240506 19:56:55.667433 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_df143abf77c545eab94fa6ed375d125f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.670181 12922 consensus_peers.cc:485] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ff3435c11854649b7daf74f3d8c33d7. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:55.668061 12520 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b28f732edd34b1286f012abf7b018fe consensus state has no leader
I20240506 19:56:55.670987 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1b285865923e44c3a4af1f276d67dec3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.671571 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b285865923e44c3a4af1f276d67dec3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.672456 14383 consensus_queue.cc:1035] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:56:55.668418 12522 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fb8bb09098d401da62ad737ffb697ad consensus state has no leader
I20240506 19:56:55.673460 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.673992 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.674444 14383 consensus_queue.cc:1035] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:55.669265 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 673dc7c83f8748a09c41d51b42214aaf
I20240506 19:56:55.676030 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 673dc7c83f8748a09c41d51b42214aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.676354 13246 tablet_replica.cc:330] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.676568 13246 raft_consensus.cc:2229] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.676714 13246 raft_consensus.cc:2258] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.676613 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1a6834c34ebb414dbedf9e1560ac7462" } new_table_name: "default.Table_9324755e6fa74275a4fd6a9a86c63ba6" modify_external_catalogs: true
I20240506 19:56:55.677393 13246 ts_tablet_manager.cc:1875] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.678216 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 673dc7c83f8748a09c41d51b42214aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.678243 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dafb661b2443f5afadc6816868bac8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:55.678903 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df143abf77c545eab94fa6ed375d125f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:55.679912 12922 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:56:55.682432 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.679749 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet 31c440555cfe4195b3ae86e1f1eb6a34 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:55.686297 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7a1c1e363dbc41dc9708971c2656dfe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.686903 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a1c1e363dbc41dc9708971c2656dfe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.670136 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6b7f6251027641e399717012c63b5ac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.689559 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b7f6251027641e399717012c63b5ac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.690187 13246 ts_tablet_manager.cc:1888] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.690418 13246 log.cc:1196] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/673dc7c83f8748a09c41d51b42214aaf
I20240506 19:56:55.690536 13245 tablet_bootstrap.cc:492] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:55.690659 13245 ts_tablet_manager.cc:1367] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20240506 19:56:55.691330 13245 raft_consensus.cc:348] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.691498 13245 raft_consensus.cc:374] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.691545 13245 raft_consensus.cc:729] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.693439 12517 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_9068971de53c4d30bbe309692dd907f3 [id=7d2da7abfa3e48f79147c8a16820ebcb] alter complete (version 1)
I20240506 19:56:55.694070 13245 consensus_queue.cc:260] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.694478 13246 ts_tablet_manager.cc:1909] T 673dc7c83f8748a09c41d51b42214aaf P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.695263 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a8531a1793404699532295164a3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.695739 13245 ts_tablet_manager.cc:1398] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:55.696732 13245 tablet_bootstrap.cc:492] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:55.699858 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 08dafb661b2443f5afadc6816868bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.700241 12962 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.700539 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.700935 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3435c11854649b7daf74f3d8c33d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.701045 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 673dc7c83f8748a09c41d51b42214aaf (table default.Table_f44e310a873e4dd28f57ab5ff4e28050 [id=59da284ab7244c98b6627a0263be2c64]) successfully deleted
I20240506 19:56:55.701282 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.701598 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bc7cfd006f4d4ca95541dab6873d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.701893 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet d7be9788df8e47d285bd932d078c6a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.702118 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e4f3439e84b5fbb2f7bab1dbd0822 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.702400 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 997ecba7fbae41178a25bf7af4d152e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.702612 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet e32a40abc4e74586af79474e4b1675c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.698204 13246 tablet_replica.cc:330] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.699540 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c440555cfe4195b3ae86e1f1eb6a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.703425 13246 raft_consensus.cc:2229] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.703768 13246 raft_consensus.cc:2258] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:55.703828 12514 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.194:59149 (request call id 135) took 792 ms (client timeout 1000 ms). Trace:
I20240506 19:56:55.704043 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeae70e0d1144ea9e3e0f8727c15e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.704533 13245 tablet_bootstrap.cc:654] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.704931 13246 ts_tablet_manager.cc:1875] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.705125 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06187c83b94d03ae4ebb67a2873cf0" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:55.705730 13249 tablet_bootstrap.cc:492] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.705811 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f56d2180b1472a8168fb3a63589065 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:49 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.705871 13249 ts_tablet_manager.cc:1367] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.390s user 0.025s sys 0.026s
I20240506 19:56:55.706100 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e50bde3564493eb7267acb42cb8592 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.706393 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet bafd3c4706f74e7da36a8f3538760f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.706578 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1160ab972124630a9be7abeefb4aa56" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:55.706663 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.706645 13249 raft_consensus.cc:348] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.706748 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb90bf767c94d73ace6411cfd31d65c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:50 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.706809 13249 raft_consensus.cc:374] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.706852 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 673dc7c83f8748a09c41d51b42214aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.706867 13249 raft_consensus.cc:729] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.705622 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0d2620e9b044d4a5393905dd527e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.707190 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7f7171d06948388bc0fdb3514202ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.707073 12815 tablet_service.cc:1514] Processing DeleteTablet for tablet 034d6309de0740d0a5806b3847e334cb with delete_type TABLET_DATA_DELETED (replaced by tablet b6fa6add23b8403ba5a0a65ac6752a5b) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.707105 13249 consensus_queue.cc:260] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.707388 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet b899fb38bfed4e02b3708f6abec80e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.707430 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet e32a40abc4e74586af79474e4b1675c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.707684 13249 ts_tablet_manager.cc:1398] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:55.707571 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet f8951bf4873e4382bcd42a52368fb376 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:55.708585 12514 rpcz_store.cc:269] 0506 19:56:54.911667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.146613 (+234946us) service_pool.cc:225] Handling call
0506 19:56:55.233357 (+ 86744us) spinlock_profiling.cc:243] Waited 85.6 ms on lock 0x616000021d54. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 00007f8feed4915d 00007f8feed49253 00007f8feba7ad41 00007f8ff366734d 00007f8ff37ee668 00007f8fea7e2786 00007f8fea7e24b5 00007f8fe6f1104e 00007f8fe6f0ff2e 00007f8fe6f16405 00007f8fe6f16d2f
0506 19:56:55.703747 (+470390us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:55.370229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:55.370332 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:55.370340 (+ 8us) write_op.cc:435] Acquired schema lock
0506 19:56:55.370342 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:55.370510 (+ 168us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:55.370523 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:55.370528 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0506 19:56:55.370651 (+ 123us) tablet.cc:699] Row locks acquired
0506 19:56:55.370653 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:55.370688 (+ 35us) write_op.cc:270] Start()
0506 19:56:55.370723 (+ 35us) write_op.cc:276] Timestamp: P: 1715025415370828 usec, L: 0
0506 19:56:55.370727 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:55.370928 (+ 201us) log.cc:842] Serialized 1674 byte log entry
0506 19:56:55.559190 (+188262us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.559296 (+ 106us) write_op.cc:301] APPLY: starting
0506 19:56:55.559303 (+ 7us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.559346 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.559377 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.559494 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.559534 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0506 19:56:55.559543 (+ 9us) write_op.cc:312] APPLY: finished
0506 19:56:55.559653 (+ 110us) log.cc:842] Serialized 41 byte log entry
0506 19:56:55.563954 (+ 4301us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.563991 (+ 37us) write_op.cc:454] Released schema lock
0506 19:56:55.564001 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.564081 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.564086 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 19:56:55.583727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:55.583821 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:55.583829 (+ 8us) write_op.cc:435] Acquired schema lock
0506 19:56:55.583831 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:55.583912 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:55.583924 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:55.583931 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:55.583978 (+ 47us) tablet.cc:699] Row locks acquired
0506 19:56:55.583981 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:55.584011 (+ 30us) write_op.cc:270] Start()
0506 19:56:55.584042 (+ 31us) write_op.cc:276] Timestamp: P: 1715025415584034 usec, L: 0
0506 19:56:55.584047 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 19:56:55.584184 (+ 137us) log.cc:842] Serialized 356 byte log entry
0506 19:56:55.650178 (+ 65994us) op_driver.cc:444] REPLICATION: finished
0506 19:56:55.650715 (+ 537us) write_op.cc:301] APPLY: starting
0506 19:56:55.650726 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:55.650752 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:55.650776 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:55.650819 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:55.650846 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:55.650854 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:55.650904 (+ 50us) log.cc:842] Serialized 23 byte log entry
0506 19:56:55.650968 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.650985 (+ 17us) write_op.cc:454] Released schema lock
0506 19:56:55.650993 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:55.651042 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:55.651045 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":36,"spinlock_wait_cycles":196946816,"child_traces":[["op",{"apply.queue_time_us":75,"apply.run_cpu_time_us":705,"apply.run_wall_time_us":4853,"mutex_wait_us":4165,"num_ops":4,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":777,"replication_time_us":188450}],["op",{"apply.queue_time_us":501,"apply.run_cpu_time_us":370,"apply.run_wall_time_us":369,"mutex_wait_us":7073,"num_ops":1,"prepare.queue_time_us":7131,"prepare.run_cpu_time_us":518,"prepare.run_wall_time_us":518,"replication_time_us":66118,"spinlock_wait_cycles":384}]]}
I20240506 19:56:55.709698 12828 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095f9fc28ff48f68064ec8318ff7b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:39 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.709887 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fb091762144f6591152680a6acd29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:36 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.712036 14590 tablet.cc:1722] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240506 19:56:55.715597 12921 consensus_peers.cc:485] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:55.715775 12921 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.715873 12921 leader_election.cc:302] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.716348 14460 raft_consensus.cc:2726] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.717067 12921 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.718278 12921 leader_election.cc:302] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.718550 14383 raft_consensus.cc:2726] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.718988 12921 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.719133 12921 leader_election.cc:302] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.719367 14383 raft_consensus.cc:2726] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.719760 12921 consensus_peers.cc:485] T 8a5217c257ff46e48c23563e496b6f08 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:55.720763 12921 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.720909 12921 leader_election.cc:302] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.721153 14383 raft_consensus.cc:2726] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.722851 14557 tablet_replica.cc:330] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.723058 14557 raft_consensus.cc:2229] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.723196 14557 raft_consensus.cc:2258] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.723735 14557 ts_tablet_manager.cc:1875] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.725317 13249 tablet_bootstrap.cc:492] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.727283 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 04738def56324a368f8182543107e32e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.733099 13244 tablet_bootstrap.cc:492] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.733561 13244 ts_tablet_manager.cc:1367] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.372s user 0.028s sys 0.020s
I20240506 19:56:55.734663 13244 raft_consensus.cc:348] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.734903 13244 raft_consensus.cc:374] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.734970 13244 raft_consensus.cc:729] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.727614 13249 tablet_bootstrap.cc:654] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.736078 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20aa06290b9a4814a006c423ccd8fc46" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:55.728929 12921 consensus_peers.cc:485] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:55.737902 12921 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.742938 13249 tablet_bootstrap.cc:492] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.743587 13249 ts_tablet_manager.cc:1367] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240506 19:56:55.735209 13244 consensus_queue.cc:260] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.745612 13244 ts_tablet_manager.cc:1398] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240506 19:56:55.746431 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]). Expected version 3 got 0
I20240506 19:56:55.747081 13244 tablet_bootstrap.cc:492] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.748420 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2c81ac92d04ddc93cc52e5c01cbc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.748600 13244 tablet_bootstrap.cc:654] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.749526 13249 raft_consensus.cc:348] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.749776 13249 raft_consensus.cc:374] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.749845 13249 raft_consensus.cc:729] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.750043 13249 consensus_queue.cc:260] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.751899 13249 ts_tablet_manager.cc:1398] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:55.752184 13249 tablet_bootstrap.cc:492] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.756268 13249 tablet_bootstrap.cc:654] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.761376 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:55.761623 14398 raft_consensus.cc:482] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.761690 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 607b3fc2c17b4793af9b84c19066ec5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.761778 14398 raft_consensus.cc:504] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.762436 14398 leader_election.cc:288] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:55.769438 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbb1c433315c4bdbac4419ef8345c075 consensus state has no leader
I20240506 19:56:55.773895 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6e0126d9854215a0e174a76c0a0391 with delete_type TABLET_DATA_DELETED (replaced by tablet 986f9c83b04f41b49e9b78fce731a90f) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.774489 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c440555cfe4195b3ae86e1f1eb6a34" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:55.775218 13058 leader_election.cc:341] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.779394 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5553b476197648cd9965dd87bbabb42f" } modify_external_catalogs: true
W20240506 19:56:55.779489 13057 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.779619 13057 leader_election.cc:302] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.780201 14398 raft_consensus.cc:2726] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.781468 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5553b476197648cd9965dd87bbabb42f" } modify_external_catalogs: true
I20240506 19:56:55.794764 13245 tablet_bootstrap.cc:492] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:55.795028 13245 ts_tablet_manager.cc:1367] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.015s
I20240506 19:56:55.795773 13245 raft_consensus.cc:348] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.797341 13245 raft_consensus.cc:374] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.797549 13245 raft_consensus.cc:729] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.797971 13245 consensus_queue.cc:260] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.799693 14557 ts_tablet_manager.cc:1888] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.799898 14557 log.cc:1196] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:56:55.800386 14557 ts_tablet_manager.cc:1909] T d9443cdbaa3c40299b66d216673c4b1a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.802146 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d9443cdbaa3c40299b66d216673c4b1a (table default.Table_456c37fe3dda48289f407577c6618ba9 [id=b164ee48c45e4826868242c03700533e]) successfully deleted
I20240506 19:56:55.802268 14557 tablet_replica.cc:330] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.802481 14557 raft_consensus.cc:2229] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.802551 13245 ts_tablet_manager.cc:1398] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:55.802735 14557 raft_consensus.cc:2258] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:55.802803 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 6b59ad067c274fddbba5f71e89480dec because the tablet was not found. No further retry: Not found: Tablet not found: 6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:55.802831 13245 tablet_bootstrap.cc:492] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:55.803412 14557 ts_tablet_manager.cc:1875] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.804360 13245 tablet_bootstrap.cc:654] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.801702 14594 tablet.cc:1722] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:55.796581 13246 ts_tablet_manager.cc:1888] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.805752 13246 log.cc:1196] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/f3a8531a1793404699532295164a3895
W20240506 19:56:55.805902 12921 consensus_peers.cc:485] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:55.805805 12781 leader_election.cc:341] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f8951bf4873e4382bcd42a52368fb376
I20240506 19:56:55.806028 12781 leader_election.cc:302] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.806244 13246 ts_tablet_manager.cc:1909] T f3a8531a1793404699532295164a3895 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.806466 14569 raft_consensus.cc:2726] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.807966 13246 tablet_replica.cc:330] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:55.808184 13246 raft_consensus.cc:2229] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.808147 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet f3a8531a1793404699532295164a3895 (table default.Table_978baf79e99f4eb8840e5030d3f7b93c [id=02d0d3a205854bb891a026ecd91c4be8]) successfully deleted
I20240506 19:56:55.808331 13246 raft_consensus.cc:2258] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.808820 13246 ts_tablet_manager.cc:1875] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:55.814980 12922 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a1160ab972124630a9be7abeefb4aa56
I20240506 19:56:55.815232 12922 leader_election.cc:302] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.815512 14383 raft_consensus.cc:2726] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.816013 12781 leader_election.cc:341] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b899fb38bfed4e02b3708f6abec80e0a
I20240506 19:56:55.822506 13249 tablet_bootstrap.cc:492] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.822876 12781 leader_election.cc:302] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.823220 14569 raft_consensus.cc:2726] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.823710 12781 leader_election.cc:341] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 63f56d2180b1472a8168fb3a63589065
I20240506 19:56:55.823858 12781 leader_election.cc:302] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.824133 14569 raft_consensus.cc:2726] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.824384 13249 ts_tablet_manager.cc:1367] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.011s
W20240506 19:56:55.824460 12781 leader_election.cc:341] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b3e50bde3564493eb7267acb42cb8592
I20240506 19:56:55.824568 12781 leader_election.cc:302] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.824781 14569 raft_consensus.cc:2726] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.825103 12781 leader_election.cc:341] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 9bb90bf767c94d73ace6411cfd31d65c
I20240506 19:56:55.825217 12781 leader_election.cc:302] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:55.825431 14569 raft_consensus.cc:2726] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.830405 13249 raft_consensus.cc:348] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.830710 13249 raft_consensus.cc:374] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.830811 13249 raft_consensus.cc:729] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.831080 13249 consensus_queue.cc:260] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.843212 13249 ts_tablet_manager.cc:1398] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240506 19:56:55.845048 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c440555cfe4195b3ae86e1f1eb6a34" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:55.845788 13249 tablet_bootstrap.cc:492] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.845973 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f9f1fa97c6434970b181f7536994782e" } new_table_name: "default.Table_0119e18037bb4223ad5b87a035d96386" modify_external_catalogs: true
I20240506 19:56:55.847537 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_99bde890b8c14fe2b57afaa1b6f6ed45" } new_table_name: "default.Table_3bd43b5745da4ad1ab82dceeec1f39b6" modify_external_catalogs: true
I20240506 19:56:55.848179 13249 tablet_bootstrap.cc:654] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.851225 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e53451477cc247e6b349f6f3be94dbd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.851949 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b" } modify_external_catalogs: true
I20240506 19:56:55.852129 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ff89b6e486b14cc0bbb0606aeeb28d1b" } modify_external_catalogs: true
I20240506 19:56:55.853260 13249 tablet_bootstrap.cc:492] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.853459 13249 ts_tablet_manager.cc:1367] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240506 19:56:55.854627 13249 raft_consensus.cc:348] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.854871 13249 raft_consensus.cc:374] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.854931 13249 raft_consensus.cc:729] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.855237 13249 consensus_queue.cc:260] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.855932 13249 ts_tablet_manager.cc:1398] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240506 19:56:55.856384 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a369e21e62444208be40d8c552b1626 consensus state has no leader
W20240506 19:56:55.859251 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e53451477cc247e6b349f6f3be94dbd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.859668 13249 tablet_bootstrap.cc:492] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.861449 13244 tablet_bootstrap.cc:492] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.861601 13244 ts_tablet_manager.cc:1367] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.015s
W20240506 19:56:55.862133 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a24a78936e4da68a4af40601b98e96 consensus state has no leader
I20240506 19:56:55.862380 13244 raft_consensus.cc:348] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.862587 13244 raft_consensus.cc:374] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.864738 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea32a023a6aa4ac1852584d211ede823" } modify_external_catalogs: true
I20240506 19:56:55.864969 13249 tablet_bootstrap.cc:654] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.864945 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ea32a023a6aa4ac1852584d211ede823" } modify_external_catalogs: true
I20240506 19:56:55.866004 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1258c1a3736349b0973af069d7d58b02" } modify_external_catalogs: true
I20240506 19:56:55.866174 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1258c1a3736349b0973af069d7d58b02" } modify_external_catalogs: true
I20240506 19:56:55.868394 13246 ts_tablet_manager.cc:1888] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.868515 13246 log.cc:1196] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/31c440555cfe4195b3ae86e1f1eb6a34
I20240506 19:56:55.868818 13246 ts_tablet_manager.cc:1909] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:55.870672 13249 tablet_bootstrap.cc:492] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.870694 14398 raft_consensus.cc:482] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.870812 13249 ts_tablet_manager.cc:1367] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240506 19:56:55.870817 14398 raft_consensus.cc:504] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.871495 14398 leader_election.cc:288] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.871637 13245 tablet_bootstrap.cc:492] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:55.871757 13245 ts_tablet_manager.cc:1367] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.001s
W20240506 19:56:55.872548 13057 leader_election.cc:341] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.872651 13057 leader_election.cc:302] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:55.873456 13057 raft_consensus.cc:2670] T 31c440555cfe4195b3ae86e1f1eb6a34 P 8a9c37db9d4f41cd93820a1c81f48d4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:55.874294 13245 raft_consensus.cc:348] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.874580 13245 raft_consensus.cc:374] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.874593 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet 5de5e3db2bf34d6ab23b58d552d2f800 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:55.874734 13245 raft_consensus.cc:729] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.875012 13245 consensus_queue.cc:260] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.875016 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_e6692a9b138044fcb858cbed4ae939ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.875592 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6692a9b138044fcb858cbed4ae939ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.875649 13245 ts_tablet_manager.cc:1398] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:55.876132 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ff7a3638004012b08c973192c055a5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:55.877365 12515 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b5bed68bde6d4275b562e5456865cea3
W20240506 19:56:55.877986 13057 leader_election.cc:341] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.862648 13244 raft_consensus.cc:729] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.878547 13244 consensus_queue.cc:260] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.879101 13244 ts_tablet_manager.cc:1398] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240506 19:56:55.879413 13244 tablet_bootstrap.cc:492] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:55.879662 13249 raft_consensus.cc:348] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.879889 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ff7a3638004012b08c973192c055a5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:55.882206 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_3d6b94b6306c4200bfde515a4b2c418c" } new_table_name: "default.Table_24285783bcc5438e9af42796d83db179" modify_external_catalogs: true
I20240506 19:56:55.887228 13244 tablet_bootstrap.cc:654] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.887984 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 971db6ac24d54efca95695ed1c90d4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.888041 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_825048dc8d55425b9dc8c3787acd53ab" } new_table_name: "default.Table_cebd347b740c4e249f4ceac983809141" modify_external_catalogs: true
I20240506 19:56:55.889384 13249 raft_consensus.cc:374] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.889493 13249 raft_consensus.cc:729] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.889747 13249 consensus_queue.cc:260] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.890280 13249 ts_tablet_manager.cc:1398] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240506 19:56:55.890566 13249 tablet_bootstrap.cc:492] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.892112 13249 tablet_bootstrap.cc:654] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:55.892102 13058 leader_election.cc:341] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.892230 13058 leader_election.cc:302] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.892621 14398 raft_consensus.cc:2726] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:55.896118 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efc41afd84a74652a82cc639bf562d99 consensus state has no leader
I20240506 19:56:55.896520 14590 tablet.cc:1722] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:55.898048 13242 ts_tablet_manager.cc:1888] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.898171 13242 log.cc:1196] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2417a535329341d8bb9f8218770f890d
I20240506 19:56:55.898449 13242 ts_tablet_manager.cc:1909] T 2417a535329341d8bb9f8218770f890d P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:55.899554 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cd37c88af8ee458ea4d03e8901f98d9f" } modify_external_catalogs: true
I20240506 19:56:55.899804 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cd37c88af8ee458ea4d03e8901f98d9f" } modify_external_catalogs: true
I20240506 19:56:55.904353 14594 tablet.cc:1722] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:55.904517 13249 tablet_bootstrap.cc:492] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:55.904659 13249 ts_tablet_manager.cc:1367] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240506 19:56:55.905212 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 912) took 2980 ms. Trace:
I20240506 19:56:55.905357 13242 rpcz_store.cc:276] 0506 19:56:52.924338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:52.924421 (+ 83us) service_pool.cc:225] Handling call
0506 19:56:55.570778 (+2646357us) ts_tablet_manager.cc:1120] Deleting tablet 2417a535329341d8bb9f8218770f890d
0506 19:56:55.570785 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.760240 (+189455us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.897928 (+137688us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.905187 (+ 7259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2646172}
I20240506 19:56:55.905928 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2417a535329341d8bb9f8218770f890d (table default.Table_c57d5247a9fc4628ac83a4fd916f2b53 [id=0db49a8d38af446f8729cfa002006438]) successfully deleted
W20240506 19:56:55.906297 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f460e0fc06a4972a2379b08fdedfc10 consensus state has no leader
I20240506 19:56:55.906558 13242 tablet_replica.cc:330] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.905427 13249 raft_consensus.cc:348] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.906989 13249 raft_consensus.cc:374] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.907045 13249 raft_consensus.cc:729] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.906859 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_aa81775ed7b04f91a8a7b6a3619fed4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.907582 13249 consensus_queue.cc:260] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:55.907666 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa81775ed7b04f91a8a7b6a3619fed4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.908161 13249 ts_tablet_manager.cc:1398] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:55.909200 13249 tablet_bootstrap.cc:492] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:55.911501 13249 tablet_bootstrap.cc:654] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.913980 14557 ts_tablet_manager.cc:1888] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.914083 14557 log.cc:1196] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:55.914381 14557 ts_tablet_manager.cc:1909] T 9df3ab9dff5148f9bb335a7209b29bbc P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:55.914784 14557 tablet_replica.cc:330] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.914969 14557 raft_consensus.cc:2229] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.915099 14557 raft_consensus.cc:2258] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.917186 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 31c440555cfe4195b3ae86e1f1eb6a34 (table default.Table_ca95166c487c4067a47b74eef95deedc [id=4fa50790af954dc899b79de2f8539532]) successfully deleted
I20240506 19:56:55.917714 13245 tablet_bootstrap.cc:492] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:55.918378 13242 raft_consensus.cc:2229] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.918569 13242 raft_consensus.cc:2258] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.918637 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4216f98cb6ac4e249ca2653518ff9e87
I20240506 19:56:55.919096 13242 ts_tablet_manager.cc:1875] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.919320 13245 tablet_bootstrap.cc:654] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:55.919426 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d630c009c25b457388c1041305861480"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.919991 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d630c009c25b457388c1041305861480 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.920334 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 9df3ab9dff5148f9bb335a7209b29bbc (table default.Table_24285783bcc5438e9af42796d83db179 [id=6b539c4e5b2444daa0f9ae05ed98ff54]) successfully deleted
I20240506 19:56:55.920830 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 4216f98cb6ac4e249ca2653518ff9e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.921505 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 4216f98cb6ac4e249ca2653518ff9e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.922140 14460 raft_consensus.cc:482] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.922262 14460 raft_consensus.cc:504] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.922817 14460 leader_election.cc:288] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:55.923032 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 4216f98cb6ac4e249ca2653518ff9e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.923148 14590 tablet.cc:1722] T 4630abf54f024da5be5b45efccc0051d P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:55.926932 13242 ts_tablet_manager.cc:1888] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.927058 13242 log.cc:1196] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b2a5a5e193f245de96cc82986e292f05
I20240506 19:56:55.927368 13242 ts_tablet_manager.cc:1909] T b2a5a5e193f245de96cc82986e292f05 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:55.927677 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 913) took 2778 ms. Trace:
I20240506 19:56:55.927763 13242 rpcz_store.cc:276] 0506 19:56:53.148962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.149031 (+ 69us) service_pool.cc:225] Handling call
0506 19:56:55.906488 (+2757457us) ts_tablet_manager.cc:1120] Deleting tablet b2a5a5e193f245de96cc82986e292f05
0506 19:56:55.906503 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.924053 (+ 17550us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.926841 (+ 2788us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.927655 (+ 814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2757215}
I20240506 19:56:55.928186 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 914) took 2703 ms. Trace:
I20240506 19:56:55.915576 14557 ts_tablet_manager.cc:1875] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.928802 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b2a5a5e193f245de96cc82986e292f05 (table default.Table_384bed2d6d984f9d88c62381e38b6321 [id=f83571d593d64e58876888bfb31ccb28]) successfully deleted
I20240506 19:56:55.929200 12522 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 5de5e3db2bf34d6ab23b58d552d2f800
I20240506 19:56:55.929687 14594 tablet.cc:1722] T 4630abf54f024da5be5b45efccc0051d P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:55.929944 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7b54e910804a4cbb8ec86b6e6f9a9246"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.930472 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b54e910804a4cbb8ec86b6e6f9a9246 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:55.930457 12922 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:56:55.932024 12522 catalog_manager.cc:6554] ReplaceTablet: tablet 5de5e3db2bf34d6ab23b58d552d2f800 deleted and replaced by tablet b156fc3b688746809597ca8bad53a3fe
I20240506 19:56:55.932973 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de5e3db2bf34d6ab23b58d552d2f800 with delete_type TABLET_DATA_DELETED (replaced by tablet b156fc3b688746809597ca8bad53a3fe) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.934769 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de5e3db2bf34d6ab23b58d552d2f800 with delete_type TABLET_DATA_DELETED (replaced by tablet b156fc3b688746809597ca8bad53a3fe) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:55.935446 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4216f98cb6ac4e249ca2653518ff9e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4216f98cb6ac4e249ca2653518ff9e87 already in progress: creating tablet
I20240506 19:56:55.939599 14557 ts_tablet_manager.cc:1888] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.932554 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de5e3db2bf34d6ab23b58d552d2f800 with delete_type TABLET_DATA_DELETED (replaced by tablet b156fc3b688746809597ca8bad53a3fe) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.936085 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffb88e0a0c4125b04f4d8d328da3f5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:55.928264 13242 rpcz_store.cc:276] 0506 19:56:53.224394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.224653 (+ 259us) service_pool.cc:225] Handling call
0506 19:56:55.928047 (+2703394us) ts_tablet_manager.cc:1120] Deleting tablet 260d7ab458794df496012cb80466f59c
0506 19:56:55.928056 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.928169 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703198}
I20240506 19:56:55.941583 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 915) took 2714 ms. Trace:
I20240506 19:56:55.941710 13242 rpcz_store.cc:276] 0506 19:56:53.227573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.227621 (+ 48us) service_pool.cc:225] Handling call
0506 19:56:55.941455 (+2713834us) ts_tablet_manager.cc:1120] Deleting tablet 0bcbf850d4914899a72f3b786d8b80b0
0506 19:56:55.941466 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.941567 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2713667}
W20240506 19:56:55.942010 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 260d7ab458794df496012cb80466f59c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 260d7ab458794df496012cb80466f59c already in progress: creating tablet
I20240506 19:56:55.942231 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 918) took 2658 ms. Trace:
W20240506 19:56:55.942322 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0bcbf850d4914899a72f3b786d8b80b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bcbf850d4914899a72f3b786d8b80b0 already in progress: creating tablet
I20240506 19:56:55.942297 13242 rpcz_store.cc:276] 0506 19:56:53.283785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.283834 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:55.942082 (+2658248us) ts_tablet_manager.cc:1120] Deleting tablet 62af97bfc4344c57b3b13ebd01f003be
0506 19:56:55.942091 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.942218 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2658089}
W20240506 19:56:55.942667 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 62af97bfc4344c57b3b13ebd01f003be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62af97bfc4344c57b3b13ebd01f003be already in progress: creating tablet
I20240506 19:56:55.943146 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 917) took 2659 ms. Trace:
I20240506 19:56:55.943285 13242 rpcz_store.cc:276] 0506 19:56:53.283689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.299450 (+ 15761us) service_pool.cc:225] Handling call
0506 19:56:55.943046 (+2643596us) ts_tablet_manager.cc:1120] Deleting tablet 467c33f012584d489c2842e3e606f485
0506 19:56:55.943054 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.943132 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2638668}
W20240506 19:56:55.943727 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 467c33f012584d489c2842e3e606f485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 467c33f012584d489c2842e3e606f485 already in progress: creating tablet
I20240506 19:56:55.944075 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 925) took 2558 ms. Trace:
I20240506 19:56:55.944195 13242 rpcz_store.cc:276] 0506 19:56:53.385432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.385630 (+ 198us) service_pool.cc:225] Handling call
0506 19:56:55.943986 (+2558356us) ts_tablet_manager.cc:1120] Deleting tablet cf6d0f3900154554b7af9338c95b5122
0506 19:56:55.943994 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.944061 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2475228}
W20240506 19:56:55.944442 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 5de5e3db2bf34d6ab23b58d552d2f800 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de5e3db2bf34d6ab23b58d552d2f800 already in progress: creating tablet
W20240506 19:56:55.944633 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet cf6d0f3900154554b7af9338c95b5122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf6d0f3900154554b7af9338c95b5122 already in progress: creating tablet
I20240506 19:56:55.944995 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 924) took 2559 ms. Trace:
W20240506 19:56:55.940801 12921 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.945164 12921 leader_election.cc:302] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.945597 14459 raft_consensus.cc:2726] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.946094 14557 log.cc:1196] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/17da01620b8b41008421b977455e0c9a
I20240506 19:56:55.946229 13242 rpcz_store.cc:276] 0506 19:56:53.385210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.390784 (+ 5574us) service_pool.cc:225] Handling call
0506 19:56:55.944904 (+2554120us) ts_tablet_manager.cc:1120] Deleting tablet b2be78cbc8264f6e8d2e05137135161c
0506 19:56:55.944912 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.944981 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2467431}
I20240506 19:56:55.946866 14557 ts_tablet_manager.cc:1909] T 17da01620b8b41008421b977455e0c9a P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:55.947283 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b2be78cbc8264f6e8d2e05137135161c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2be78cbc8264f6e8d2e05137135161c already in progress: creating tablet
I20240506 19:56:55.947652 13242 tablet_replica.cc:330] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.947924 13242 raft_consensus.cc:2229] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.949162 12847 raft_consensus.cc:3037] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:55.949800 13242 raft_consensus.cc:2258] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.950765 13242 ts_tablet_manager.cc:1875] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.952616 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 17da01620b8b41008421b977455e0c9a (table default.Table_e43a57ef696d4d749ff2f11fe74f999b [id=a445760bb4514251b6ff954ca946cd59]) successfully deleted
I20240506 19:56:55.958839 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b9e1af536a204e51a898c9821976d7ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.959406 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9e1af536a204e51a898c9821976d7ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.961171 14557 tablet_replica.cc:330] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.961517 14557 raft_consensus.cc:2229] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.961858 14557 raft_consensus.cc:2258] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.962600 14557 ts_tablet_manager.cc:1875] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.962750 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9b101aa14e0a4b1cbb4aefcc7dc75d2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.962926 14398 raft_consensus.cc:482] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:55.963060 14398 raft_consensus.cc:504] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:55.963479 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b101aa14e0a4b1cbb4aefcc7dc75d2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.963692 14398 leader_election.cc:288] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:55.964532 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b011e94e2b47d7a51d6816576ea124" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:55.964531 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b011e94e2b47d7a51d6816576ea124" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:55.965266 13058 leader_election.cc:341] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.966256 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7ac353c6b9e541f181e20ad72775899a
W20240506 19:56:55.966351 13057 leader_election.cc:341] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:55.966454 13057 leader_election.cc:302] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:55.967113 14398 raft_consensus.cc:2726] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:55.967133 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ac4cdb7dd4e44b45963cf93eaefe23a6
I20240506 19:56:55.967437 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac353c6b9e541f181e20ad72775899a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.968102 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.968526 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:55.968533 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac353c6b9e541f181e20ad72775899a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.968750 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:55.968784 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac353c6b9e541f181e20ad72775899a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:55.971335 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_110baf1a2d08406abaddd6e76dd187a4" } new_table_name: "default.Table_42f6118c60d04d7aa217e868cb38a562" modify_external_catalogs: true
W20240506 19:56:55.975812 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7ac353c6b9e541f181e20ad72775899a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ac353c6b9e541f181e20ad72775899a already in progress: creating tablet
W20240506 19:56:55.976243 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4cdb7dd4e44b45963cf93eaefe23a6 already in progress: creating tablet
I20240506 19:56:55.979234 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b5b09cfa481b48eba1086b706fb28557"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.980098 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5b09cfa481b48eba1086b706fb28557 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.983781 13242 ts_tablet_manager.cc:1888] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.983916 13242 log.cc:1196] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/30ad527c3f2a4b53affa928e149db1df
I20240506 19:56:55.984418 12513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0c23ca41c31d456e89fdd0046e87d6f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:55.984879 14557 ts_tablet_manager.cc:1888] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:55.984973 14557 log.cc:1196] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2a7c61244620487bbbac26dcdbf8a38c
W20240506 19:56:55.984984 12513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c23ca41c31d456e89fdd0046e87d6f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.985443 13242 ts_tablet_manager.cc:1909] T 30ad527c3f2a4b53affa928e149db1df P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:55.985724 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 937) took 2416 ms. Trace:
I20240506 19:56:55.985790 13242 rpcz_store.cc:276] 0506 19:56:53.568840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.571124 (+ 2284us) service_pool.cc:225] Handling call
0506 19:56:55.947600 (+2376476us) ts_tablet_manager.cc:1120] Deleting tablet 30ad527c3f2a4b53affa928e149db1df
0506 19:56:55.947612 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.973180 (+ 25568us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.983669 (+ 10489us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:55.985707 (+ 2038us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376258}
I20240506 19:56:55.986209 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 30ad527c3f2a4b53affa928e149db1df (table default.Table_052f7d1eab54464d8743bd859c9f5bad [id=cdb5ce64abc74bb3bf60d7aa727c515f]) successfully deleted
I20240506 19:56:55.986594 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 944) took 2382 ms. Trace:
I20240506 19:56:55.986670 13242 rpcz_store.cc:276] 0506 19:56:53.604412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.604588 (+ 176us) service_pool.cc:225] Handling call
0506 19:56:55.986478 (+2381890us) ts_tablet_manager.cc:1120] Deleting tablet f1a792bf9b774b8f8cc5491605f504ac
0506 19:56:55.986487 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.986579 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381710}
I20240506 19:56:55.986754 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1f73021793a44d038dfb378928d261d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:55.987046 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f1a792bf9b774b8f8cc5491605f504ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a792bf9b774b8f8cc5491605f504ac already in progress: creating tablet
I20240506 19:56:55.985229 14557 ts_tablet_manager.cc:1909] T 2a7c61244620487bbbac26dcdbf8a38c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:55.987248 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f73021793a44d038dfb378928d261d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:55.989439 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2a7c61244620487bbbac26dcdbf8a38c (table default.Table_7b7dd900dc5b478280237dfd98e0f303 [id=2ec62daef8074bedbd19401d8245b14a]) successfully deleted
I20240506 19:56:55.990263 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 945) took 2385 ms. Trace:
I20240506 19:56:55.990377 13242 rpcz_store.cc:276] 0506 19:56:53.604536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.614926 (+ 10390us) service_pool.cc:225] Handling call
0506 19:56:55.990117 (+2375191us) ts_tablet_manager.cc:1120] Deleting tablet 8bd1b51c240a4b1aad65839679f7bba7
0506 19:56:55.990131 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:55.990244 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2374751}
W20240506 19:56:55.990882 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 8bd1b51c240a4b1aad65839679f7bba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bd1b51c240a4b1aad65839679f7bba7 already in progress: creating tablet
W20240506 19:56:55.991858 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 1615f6167aae4dc096b9f563ab5e6215 because the tablet was not found. No further retry: Not found: Tablet not found: 1615f6167aae4dc096b9f563ab5e6215
I20240506 19:56:55.992303 13242 tablet_replica.cc:330] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:55.992600 13242 raft_consensus.cc:2229] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.992772 13242 raft_consensus.cc:2258] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.993541 14557 tablet_replica.cc:330] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:55.993813 14557 raft_consensus.cc:2229] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:55.994046 14557 raft_consensus.cc:2258] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:55.994647 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:55.994668 14557 ts_tablet_manager.cc:1875] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.995131 12516 catalog_manager.cc:5484] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20240506 19:56:55.995393 13242 ts_tablet_manager.cc:1875] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:55.995585 12516 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:55.995755 13244 tablet_bootstrap.cc:492] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:55.995874 13244 ts_tablet_manager.cc:1367] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.117s user 0.006s sys 0.000s
I20240506 19:56:55.996600 13244 raft_consensus.cc:348] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.996814 13244 raft_consensus.cc:374] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:55.996871 13244 raft_consensus.cc:729] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:55.997068 13244 consensus_queue.cc:260] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:55.998919 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca87343f5914e90a1810c62ba87c51a (DEFAULT_TABLE table=default.Table_9ab15c3c5575442181350680a86a5370 [id=7f170fa8d27a423f977115e24d56b038]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:55.999392 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca87343f5914e90a1810c62ba87c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:55.999958 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca87343f5914e90a1810c62ba87c51a (DEFAULT_TABLE table=default.Table_9ab15c3c5575442181350680a86a5370 [id=7f170fa8d27a423f977115e24d56b038]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.000376 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca87343f5914e90a1810c62ba87c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.002019 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca87343f5914e90a1810c62ba87c51a (DEFAULT_TABLE table=default.Table_9ab15c3c5575442181350680a86a5370 [id=7f170fa8d27a423f977115e24d56b038]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.002511 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca87343f5914e90a1810c62ba87c51a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:56.002965 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d8a94ecb64b4b64889b3585e64fff5a consensus state has no leader
I20240506 19:56:56.004280 13090 tablet_service.cc:1467] Processing CreateTablet for tablet dbc62694b412441ea8a406cef0e4ea3b (DEFAULT_TABLE table=default.Table_c4ab6be6ef2d46bdb824d78a7eea1f20 [id=f4cd8e862ab54bf3b10336065c65b7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.004987 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc62694b412441ea8a406cef0e4ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.009572 12828 tablet_service.cc:1467] Processing CreateTablet for tablet dbc62694b412441ea8a406cef0e4ea3b (DEFAULT_TABLE table=default.Table_c4ab6be6ef2d46bdb824d78a7eea1f20 [id=f4cd8e862ab54bf3b10336065c65b7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.009956 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc62694b412441ea8a406cef0e4ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.013677 12952 tablet_service.cc:1467] Processing CreateTablet for tablet dbc62694b412441ea8a406cef0e4ea3b (DEFAULT_TABLE table=default.Table_c4ab6be6ef2d46bdb824d78a7eea1f20 [id=f4cd8e862ab54bf3b10336065c65b7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.014113 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc62694b412441ea8a406cef0e4ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.018834 14569 raft_consensus.cc:482] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.019001 14569 raft_consensus.cc:504] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.019894 14569 leader_election.cc:288] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.021538 12781 leader_election.cc:341] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 607b3fc2c17b4793af9b84c19066ec5e
I20240506 19:56:56.021677 13245 tablet_bootstrap.cc:492] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:56:56.021754 12783 leader_election.cc:341] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 607b3fc2c17b4793af9b84c19066ec5e
I20240506 19:56:56.021812 13245 ts_tablet_manager.cc:1367] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.011s
I20240506 19:56:56.021829 12783 leader_election.cc:302] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.022433 14569 raft_consensus.cc:2726] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.022565 13245 raft_consensus.cc:348] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.022821 13245 raft_consensus.cc:374] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.022908 13245 raft_consensus.cc:729] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.023164 13245 consensus_queue.cc:260] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.029806 13245 ts_tablet_manager.cc:1398] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:56.030422 13245 tablet_bootstrap.cc:492] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.038378 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 689901ddfa9f492f98b67daf000a28ae (DEFAULT_TABLE table=default.Table_82ad0ca5d8e2425b98dc9d534a3436cc [id=e4c63adb11504619b37bdc6c1e025c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.038847 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689901ddfa9f492f98b67daf000a28ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.042608 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 689901ddfa9f492f98b67daf000a28ae (DEFAULT_TABLE table=default.Table_82ad0ca5d8e2425b98dc9d534a3436cc [id=e4c63adb11504619b37bdc6c1e025c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.042647 12960 tablet_service.cc:1467] Processing CreateTablet for tablet c41c5ac8267a466e99ed3d88170d6f25 (DEFAULT_TABLE table=default.Table_829c185590b0466f86672ee2ce1d349f [id=2f5efbdf6ae84721b52f3e9396ea42f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.042647 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 803d85088fba469d9175dd94330cac0f (DEFAULT_TABLE table=default.Table_5c27619f39744d3aa10bfc80a3397da5 [id=8879a2545913487dafc8603951a8e8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.042713 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 803d85088fba469d9175dd94330cac0f (DEFAULT_TABLE table=default.Table_5c27619f39744d3aa10bfc80a3397da5 [id=8879a2545913487dafc8603951a8e8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.043046 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803d85088fba469d9175dd94330cac0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.043385 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41c5ac8267a466e99ed3d88170d6f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.043399 13244 ts_tablet_manager.cc:1398] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20240506 19:56:56.043720 13244 tablet_bootstrap.cc:492] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.045213 13244 tablet_bootstrap.cc:654] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.045730 13085 tablet_service.cc:1467] Processing CreateTablet for tablet c41c5ac8267a466e99ed3d88170d6f25 (DEFAULT_TABLE table=default.Table_829c185590b0466f86672ee2ce1d349f [id=2f5efbdf6ae84721b52f3e9396ea42f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.046108 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41c5ac8267a466e99ed3d88170d6f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.047066 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803d85088fba469d9175dd94330cac0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.049616 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689901ddfa9f492f98b67daf000a28ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.049705 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 689901ddfa9f492f98b67daf000a28ae (DEFAULT_TABLE table=default.Table_82ad0ca5d8e2425b98dc9d534a3436cc [id=e4c63adb11504619b37bdc6c1e025c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.050036 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689901ddfa9f492f98b67daf000a28ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.056499 12817 tablet_service.cc:1467] Processing CreateTablet for tablet c41c5ac8267a466e99ed3d88170d6f25 (DEFAULT_TABLE table=default.Table_829c185590b0466f86672ee2ce1d349f [id=2f5efbdf6ae84721b52f3e9396ea42f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.057092 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41c5ac8267a466e99ed3d88170d6f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.057597 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 803d85088fba469d9175dd94330cac0f (DEFAULT_TABLE table=default.Table_5c27619f39744d3aa10bfc80a3397da5 [id=8879a2545913487dafc8603951a8e8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.058010 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803d85088fba469d9175dd94330cac0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.059370 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_7881939105214872a57c956985a14360" } new_table_name: "default.Table_16fd6d89a90c41eaaa2e17854af6470e" modify_external_catalogs: true
I20240506 19:56:56.061691 14459 raft_consensus.cc:482] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.061889 14459 raft_consensus.cc:504] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.062812 14459 leader_election.cc:288] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.064261 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_cc43c588b4804f9aa2bbadd0d65ce064" } new_table_name: "default.Table_bb460b3bc3c6433aa0d6abe2f3eea2e2" modify_external_catalogs: true
W20240506 19:56:56.064707 12922 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 116df9bd6ab04a30ba09861e765d7267
I20240506 19:56:56.065408 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116df9bd6ab04a30ba09861e765d7267" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.066097 12921 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.066237 12921 leader_election.cc:302] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.068003 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet d842127c078f4d02bca0c63531e72c0d
I20240506 19:56:56.069480 14383 raft_consensus.cc:2726] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.070802 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet e1d7a850a3e644c3a38ab224efcc8af5 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:56.072302 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet d842127c078f4d02bca0c63531e72c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.073032 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc57eb8ac864bfa803908412530e7a3 (DEFAULT_TABLE table=default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.073558 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc57eb8ac864bfa803908412530e7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.074846 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet d842127c078f4d02bca0c63531e72c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:56.075575 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 55864.8us
I20240506 19:56:56.075930 13245 tablet_bootstrap.cc:654] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.079433 13101 tablet_service.cc:1467] Processing CreateTablet for tablet e0e05e5b413c4e899dff9d4dde01d0ca (DEFAULT_TABLE table=default.Table_d0df8358053749a49c25b73041023ac1 [id=351289efb08e425da033ce80530da032]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.079802 13090 tablet_service.cc:1467] Processing CreateTablet for tablet b04f4a9ca51d452789f5dc4bd36f5c6f (DEFAULT_TABLE table=default.Table_2a09a031cefc4757938ad6047310bc72 [id=09e89c3a03164e878e6182dddb162d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.080188 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc57eb8ac864bfa803908412530e7a3 (DEFAULT_TABLE table=default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.080870 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet d842127c078f4d02bca0c63531e72c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:55 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.081087 12825 tablet_service.cc:1467] Processing CreateTablet for tablet e0e05e5b413c4e899dff9d4dde01d0ca (DEFAULT_TABLE table=default.Table_d0df8358053749a49c25b73041023ac1 [id=351289efb08e425da033ce80530da032]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.081508 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05e5b413c4e899dff9d4dde01d0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.081563 12827 tablet_service.cc:1467] Processing CreateTablet for tablet b04f4a9ca51d452789f5dc4bd36f5c6f (DEFAULT_TABLE table=default.Table_2a09a031cefc4757938ad6047310bc72 [id=09e89c3a03164e878e6182dddb162d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.081964 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04f4a9ca51d452789f5dc4bd36f5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.081970 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 46916e778bf34188959e1cfff084bffc (table default.Table_fbd2d442ecdb4935bdff3a94b68cc073 [id=df889494c5f34752bf16bde1b5a80964]). Expected version 2 got 0
I20240506 19:56:56.082199 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:56.082341 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:56.082509 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet e1b011e94e2b47d7a51d6816576ea124 (table default.Table_ef03c2589fde409f95b535d84b66bf29 [id=e8385d8ad6714acf91d8cb6b0de6469c]). Expected version 2 got 0
I20240506 19:56:56.082626 12517 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 20aa06290b9a4814a006c423ccd8fc46 (table default.Table_e050c5de47634fe69b12743c8b30d5e9 [id=75db5850c6074469bfd66e8d26699e05]). Expected version 2 got 0
I20240506 19:56:56.081264 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc57eb8ac864bfa803908412530e7a3 (DEFAULT_TABLE table=default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.085489 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ce89088ef02b4c90bcfd607e38ba1608" } modify_external_catalogs: true
I20240506 19:56:56.085665 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc57eb8ac864bfa803908412530e7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.085690 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ce89088ef02b4c90bcfd607e38ba1608" } modify_external_catalogs: true
I20240506 19:56:56.085733 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ff7a3638004012b08c973192c055a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.093971 13246 tablet_replica.cc:330] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:56.094184 13246 raft_consensus.cc:2229] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.094344 13246 raft_consensus.cc:2258] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.094478 14398 raft_consensus.cc:482] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.094658 14398 raft_consensus.cc:504] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.094938 13246 ts_tablet_manager.cc:1875] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.095551 14398 leader_election.cc:288] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.096232 13244 tablet_bootstrap.cc:492] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.096364 13244 ts_tablet_manager.cc:1367] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240506 19:56:56.096477 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46916e778bf34188959e1cfff084bffc" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.097113 13244 raft_consensus.cc:348] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.097316 13244 raft_consensus.cc:374] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.097380 13244 raft_consensus.cc:729] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:56.097460 13058 leader_election.cc:341] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.097438 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46916e778bf34188959e1cfff084bffc" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.097579 13244 consensus_queue.cc:260] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.097991 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet d842127c078f4d02bca0c63531e72c0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d842127c078f4d02bca0c63531e72c0d already in progress: creating tablet
W20240506 19:56:56.102588 12921 consensus_peers.cc:485] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:56.102970 13246 ts_tablet_manager.cc:1888] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:56.103071 13057 leader_election.cc:341] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.103219 13057 leader_election.cc:302] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.103296 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05e5b413c4e899dff9d4dde01d0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.103771 14398 raft_consensus.cc:2726] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.106360 12953 tablet_service.cc:1467] Processing CreateTablet for tablet b04f4a9ca51d452789f5dc4bd36f5c6f (DEFAULT_TABLE table=default.Table_2a09a031cefc4757938ad6047310bc72 [id=09e89c3a03164e878e6182dddb162d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.106590 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8c32c6726a7f4aa2867ec4f2594083d5" } new_table_name: "default.Table_1c4853e227ec4b02ab3929603dbba13e" modify_external_catalogs: true
I20240506 19:56:56.106765 12962 tablet_service.cc:1467] Processing CreateTablet for tablet e0e05e5b413c4e899dff9d4dde01d0ca (DEFAULT_TABLE table=default.Table_d0df8358053749a49c25b73041023ac1 [id=351289efb08e425da033ce80530da032]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.106768 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0ac966a105410b8d1b965eb2ad6082 (DEFAULT_TABLE table=default.Table_ad11ec97e2384012981d0daae3a8adbd [id=6921d75de60f46e087367fd1b28ce12d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.106982 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d1f01c0edb4f6b8a621c1643ca1afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.107169 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0ac966a105410b8d1b965eb2ad6082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.107143 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26e6b8115644cabe6532c7be1bbe95 with delete_type TABLET_DATA_DELETED (replaced by tablet a152a0f6a9ed4aa5b74fe5a53688331d) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.107328 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet da06187c83b94d03ae4ebb67a2873cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet d1ebd7b994584b2bab6d27f493deab92) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.107481 12961 tablet_service.cc:1514] Processing DeleteTablet for tablet a1160ab972124630a9be7abeefb4aa56 with delete_type TABLET_DATA_DELETED (replaced by tablet 44ce8749846847108edf5f554139cbd9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.108204 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0ac966a105410b8d1b965eb2ad6082 (DEFAULT_TABLE table=default.Table_ad11ec97e2384012981d0daae3a8adbd [id=6921d75de60f46e087367fd1b28ce12d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.109025 12957 tablet_service.cc:1467] Processing CreateTablet for tablet a47b76562ce04a72b73b5055afb04f0c (DEFAULT_TABLE table=default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.109586 13096 tablet_service.cc:1467] Processing CreateTablet for tablet a47b76562ce04a72b73b5055afb04f0c (DEFAULT_TABLE table=default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.110541 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 9eeabb650bc04bc0a67a98afb10c17e3 (DEFAULT_TABLE table=default.Table_251bb78261d6402bbf113e3110a57ba2 [id=05680d808e924fcfbe8d79454a3ded17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.110944 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eeabb650bc04bc0a67a98afb10c17e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.111150 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47b76562ce04a72b73b5055afb04f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.112411 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b35bb26efd3c4ca08971e6c1185a6c0b" } new_table_name: "default.Table_86e2e3b68d354e02b29118d063456d4a" modify_external_catalogs: true
I20240506 19:56:56.113837 12824 tablet_service.cc:1467] Processing CreateTablet for tablet a47b76562ce04a72b73b5055afb04f0c (DEFAULT_TABLE table=default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.114363 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47b76562ce04a72b73b5055afb04f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.122395 14459 consensus_queue.cc:1035] T 6352d2c630204df3a24e299e167f92d4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:56.122772 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04f4a9ca51d452789f5dc4bd36f5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.125571 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05e5b413c4e899dff9d4dde01d0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.125830 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0ac966a105410b8d1b965eb2ad6082 (DEFAULT_TABLE table=default.Table_ad11ec97e2384012981d0daae3a8adbd [id=6921d75de60f46e087367fd1b28ce12d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.126256 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0ac966a105410b8d1b965eb2ad6082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.129931 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0f648dd8cb4a4b88982f126032caf2f5" } modify_external_catalogs: true
I20240506 19:56:56.131134 14436 raft_consensus.cc:482] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.131275 14436 raft_consensus.cc:504] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:56.131428 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0ac966a105410b8d1b965eb2ad6082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.131919 14436 leader_election.cc:288] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:56:56.132617 13244 ts_tablet_manager.cc:1398] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
W20240506 19:56:56.132679 12781 leader_election.cc:341] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:56.132923 13244 tablet_bootstrap.cc:492] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.133721 13096 tablet_service.cc:1467] Processing CreateTablet for tablet 9eeabb650bc04bc0a67a98afb10c17e3 (DEFAULT_TABLE table=default.Table_251bb78261d6402bbf113e3110a57ba2 [id=05680d808e924fcfbe8d79454a3ded17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.134151 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eeabb650bc04bc0a67a98afb10c17e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.135516 13249 tablet_bootstrap.cc:492] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.135661 13249 ts_tablet_manager.cc:1367] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.227s user 0.015s sys 0.026s
I20240506 19:56:56.136343 13249 raft_consensus.cc:348] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.136525 13249 raft_consensus.cc:374] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.136584 13249 raft_consensus.cc:729] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.137277 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47b76562ce04a72b73b5055afb04f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.141458 13244 tablet_bootstrap.cc:654] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.145435 14459 raft_consensus.cc:482] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.145574 14459 raft_consensus.cc:504] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.145790 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 9eeabb650bc04bc0a67a98afb10c17e3 (DEFAULT_TABLE table=default.Table_251bb78261d6402bbf113e3110a57ba2 [id=05680d808e924fcfbe8d79454a3ded17]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.146137 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eeabb650bc04bc0a67a98afb10c17e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.146173 14459 leader_election.cc:288] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.146817 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04f4a9ca51d452789f5dc4bd36f5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.147397 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d1f01c0edb4f6b8a621c1643ca1afb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.147918 12921 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.148775 13249 consensus_queue.cc:260] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.153367 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc57eb8ac864bfa803908412530e7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.153899 13249 ts_tablet_manager.cc:1398] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240506 19:56:56.103083 13246 log.cc:1196] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/c1ff7a3638004012b08c973192c055a5
I20240506 19:56:56.154568 13246 ts_tablet_manager.cc:1909] T c1ff7a3638004012b08c973192c055a5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:56.156941 12922 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:56:56.157056 12922 leader_election.cc:302] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.157490 14459 raft_consensus.cc:2726] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:56.159232 12783 leader_election.cc:341] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:56.159435 12783 leader_election.cc:302] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.159752 14569 raft_consensus.cc:2726] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.160113 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_0f648dd8cb4a4b88982f126032caf2f5" } modify_external_catalogs: true
I20240506 19:56:56.163053 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_02edd5e110a34ea292b65d41b0963109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.163622 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02edd5e110a34ea292b65d41b0963109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.165256 14617 tablet.cc:1722] T 176c61208b1c4518820a829d9f5c0a6f P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:56.167089 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a0d74db25f05423f8b5ae3b82c7a54b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:56.166916 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9db7aa6cc91143a9be731a84793c16dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.168709 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db7aa6cc91143a9be731a84793c16dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:56.170446 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0d74db25f05423f8b5ae3b82c7a54b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.171339 14569 raft_consensus.cc:482] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.171502 14569 raft_consensus.cc:504] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.172223 14569 leader_election.cc:288] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.174338 12781 leader_election.cc:341] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: bd2c81ac92d04ddc93cc52e5c01cbc60
I20240506 19:56:56.175472 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet c1ff7a3638004012b08c973192c055a5 (table default.Table_e5114df3e3df4ea1bfa99b29bb3dcbdc [id=0f37f31664bf47f1a74d2c82f8767eff]) successfully deleted
W20240506 19:56:56.177345 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 07a24a78936e4da68a4af40601b98e96 (table default.Table_3bd43b5745da4ad1ab82dceeec1f39b6 [id=38092479ce4249f5aed0ab679d4374ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 07a24a78936e4da68a4af40601b98e96 consensus state has no leader
I20240506 19:56:56.177784 13249 tablet_bootstrap.cc:492] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.181650 13249 tablet_bootstrap.cc:654] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.181959 12783 leader_election.cc:341] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bd2c81ac92d04ddc93cc52e5c01cbc60
I20240506 19:56:56.182152 12783 leader_election.cc:302] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.182433 14569 raft_consensus.cc:2726] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:56.190367 12521 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b904072d89c6486fa3ca1557220f0b52 consensus state has no leader
W20240506 19:56:56.193670 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 054d48760d684adeb8196b0532b07101 consensus state has no leader
I20240506 19:56:56.194100 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_111e3f9425804b4ca0bf599cc9377867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.194667 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_111e3f9425804b4ca0bf599cc9377867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.198135 13249 tablet_bootstrap.cc:492] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.198340 13249 ts_tablet_manager.cc:1367] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20240506 19:56:56.198418 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:56.198846 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9d0d827d732e4261a7a8edb549c1c8e3 (table default.Table_9eea3fa543d74708aed9bf6f5e378d89 [id=25076b7bd89a4c3bbb205acaab09690a]). Expected version 1 got 0
I20240506 19:56:56.199123 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:56.199317 12521 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]). Expected version 1 got 0
I20240506 19:56:56.207080 13245 tablet_bootstrap.cc:492] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.213085 14590 tablet.cc:1722] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:56.216275 14594 tablet.cc:1722] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:56.219200 13245 ts_tablet_manager.cc:1367] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.189s user 0.003s sys 0.002s
I20240506 19:56:56.220068 13245 raft_consensus.cc:348] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.220446 13245 raft_consensus.cc:374] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.220561 13245 raft_consensus.cc:729] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.220837 13245 consensus_queue.cc:260] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.205379 13249 raft_consensus.cc:348] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.221100 13249 raft_consensus.cc:374] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.221184 13249 raft_consensus.cc:729] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.221438 13249 consensus_queue.cc:260] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.221709 13245 ts_tablet_manager.cc:1398] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:56.225658 13249 ts_tablet_manager.cc:1398] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240506 19:56:56.226085 13249 tablet_bootstrap.cc:492] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.239022 13249 tablet_bootstrap.cc:654] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.246174 13249 tablet_bootstrap.cc:492] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.250082 14398 raft_consensus.cc:482] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.250237 14398 raft_consensus.cc:504] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.250525 13245 tablet_bootstrap.cc:492] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.250828 14398 leader_election.cc:288] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.251271 13249 ts_tablet_manager.cc:1367] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240506 19:56:56.251595 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e04e93a5543f9934077006eaf2ba9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.251961 13245 tablet_bootstrap.cc:654] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.252130 13057 leader_election.cc:341] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.252301 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ddae4b0d8d0943d99f54904266f293d4
I20240506 19:56:56.254035 13249 raft_consensus.cc:348] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.254225 13249 raft_consensus.cc:374] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:56.254236 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92329da0f9ec4d2e982efe5e9b94c5d6 consensus state has no leader
I20240506 19:56:56.254267 13249 raft_consensus.cc:729] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.255738 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e04e93a5543f9934077006eaf2ba9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.257328 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 60edeed6ae3b48e2b9f5341d7e7af01d
W20240506 19:56:56.259095 13058 leader_election.cc:341] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.259483 13249 consensus_queue.cc:260] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.259583 13058 leader_election.cc:302] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.260104 14398 raft_consensus.cc:2726] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.260439 13249 ts_tablet_manager.cc:1398] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:56:56.260879 13169 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
I20240506 19:56:56.261855 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet ddae4b0d8d0943d99f54904266f293d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.264911 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet ddae4b0d8d0943d99f54904266f293d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.265265 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 60edeed6ae3b48e2b9f5341d7e7af01d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.266007 13249 tablet_bootstrap.cc:492] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.266325 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 60edeed6ae3b48e2b9f5341d7e7af01d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.267547 14590 tablet.cc:1722] T 5751124adb5b40f59e8d79a4605ae8c4 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:56.271010 14557 ts_tablet_manager.cc:1888] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.273401 14594 tablet.cc:1722] T 5751124adb5b40f59e8d79a4605ae8c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:56.273931 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet ddae4b0d8d0943d99f54904266f293d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:56.275489 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ddae4b0d8d0943d99f54904266f293d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddae4b0d8d0943d99f54904266f293d4 already in progress: creating tablet
I20240506 19:56:56.276633 13244 tablet_bootstrap.cc:492] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.276805 13244 ts_tablet_manager.cc:1367] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.000s
I20240506 19:56:56.277549 14630 tablet.cc:1722] T dcb357a7eb2147a79cc69d024bd35269 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:56.278156 13244 raft_consensus.cc:348] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.278362 13244 raft_consensus.cc:374] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.278411 13244 raft_consensus.cc:729] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.285789 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 60edeed6ae3b48e2b9f5341d7e7af01d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.286545 13244 consensus_queue.cc:260] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.287214 13244 ts_tablet_manager.cc:1398] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
W20240506 19:56:56.288179 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 60edeed6ae3b48e2b9f5341d7e7af01d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60edeed6ae3b48e2b9f5341d7e7af01d already in progress: creating tablet
I20240506 19:56:56.294164 14569 raft_consensus.cc:482] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.294337 14569 raft_consensus.cc:504] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.294571 13242 ts_tablet_manager.cc:1888] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.294683 13242 log.cc:1196] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/9e3d13322b4a4a2db3b6d425679f68ed
I20240506 19:56:56.295003 13242 ts_tablet_manager.cc:1909] T 9e3d13322b4a4a2db3b6d425679f68ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.295051 14569 leader_election.cc:288] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.296208 12781 leader_election.cc:341] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 4acde41b2c03440a89243aae008779e1
I20240506 19:56:56.272749 14460 raft_consensus.cc:482] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.275714 14590 tablet.cc:1722] T dcb357a7eb2147a79cc69d024bd35269 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:56.298709 14557 log.cc:1196] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2b3268094cbf481698cb373c1bf6dfba
I20240506 19:56:56.299093 14557 ts_tablet_manager.cc:1909] T 2b3268094cbf481698cb373c1bf6dfba P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.299921 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]). Expected version 3 got 0
I20240506 19:56:56.297669 13249 tablet_bootstrap.cc:654] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.300835 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_4b829083fa8c4c29b7b0c730361bb0d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.301544 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b829083fa8c4c29b7b0c730361bb0d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.301563 14460 raft_consensus.cc:504] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.302232 14460 leader_election.cc:288] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.303592 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_68c8729d5e134c10be8d88e246c4e00d" } new_table_name: "default.Table_d0265b889b7d4b35a1e6df3bcdfbe208" modify_external_catalogs: true
I20240506 19:56:56.304924 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_20af0dc96de14fdf9a880466506fde22" } modify_external_catalogs: true
I20240506 19:56:56.305086 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_20af0dc96de14fdf9a880466506fde22" } modify_external_catalogs: true
I20240506 19:56:56.305773 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4e97d10ae54e558920d116d8fa0ba5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:56.305992 13120 raft_consensus.cc:2454] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:56.306386 12922 leader_election.cc:302] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:56.306481 12520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c05d8864f0ae4794a8a633bd250e7569"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:56.306685 14460 raft_consensus.cc:2781] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:56.306756 14460 raft_consensus.cc:482] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:56.306813 14460 raft_consensus.cc:3037] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:56.307032 12520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c05d8864f0ae4794a8a633bd250e7569 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.308450 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_946004ac61bd4421abf9cdfa64f35460"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.308907 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_946004ac61bd4421abf9cdfa64f35460 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.310182 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_12084251f64140ce8b5be5e06ed93ed3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.310596 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12084251f64140ce8b5be5e06ed93ed3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.311295 14460 raft_consensus.cc:504] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.311959 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_803e3127dd644ef7a78b9e24616a7d66" } modify_external_catalogs: true
I20240506 19:56:56.312119 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_803e3127dd644ef7a78b9e24616a7d66" } modify_external_catalogs: true
I20240506 19:56:56.312345 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4e97d10ae54e558920d116d8fa0ba5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:56.312533 13120 raft_consensus.cc:3037] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:56.314044 12783 leader_election.cc:341] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 4acde41b2c03440a89243aae008779e1
I20240506 19:56:56.314239 12783 leader_election.cc:302] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.314518 14569 raft_consensus.cc:2726] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.315541 13120 raft_consensus.cc:2454] T 6f4e97d10ae54e558920d116d8fa0ba5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:56.315979 12922 leader_election.cc:302] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:56.316430 14460 leader_election.cc:288] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.316607 14460 raft_consensus.cc:2781] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:56.316557 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_51c8c97056294104afce55e84e26a550" } modify_external_catalogs: true
I20240506 19:56:56.316709 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_51c8c97056294104afce55e84e26a550" } modify_external_catalogs: true
I20240506 19:56:56.316833 14460 raft_consensus.cc:686] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:56.317734 14460 consensus_queue.cc:237] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.319906 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1059) took 1111 ms. Trace:
I20240506 19:56:56.322005 14557 rpcz_store.cc:276] 0506 19:56:55.208711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.210738 (+ 2027us) service_pool.cc:225] Handling call
0506 19:56:55.993475 (+782737us) ts_tablet_manager.cc:1120] Deleting tablet 2b3268094cbf481698cb373c1bf6dfba
0506 19:56:55.993489 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.267843 (+274354us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.270887 (+ 3044us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.319889 (+ 49002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":782608}
I20240506 19:56:56.322715 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2b3268094cbf481698cb373c1bf6dfba (table default.Table_715977f202b24ca7a07c82ef612e23bb [id=ef86771f67d14d9090bcc7722a834478]) successfully deleted
I20240506 19:56:56.325742 14557 tablet_replica.cc:330] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.326007 14557 raft_consensus.cc:2229] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:56.326345 14557 pending_rounds.cc:62] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:56.326444 14557 pending_rounds.cc:68] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744090618376192 op_type: NO_OP noop_request { }
I20240506 19:56:56.326597 14557 raft_consensus.cc:2866] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:56.326709 14557 raft_consensus.cc:2258] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.327266 14557 ts_tablet_manager.cc:1875] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.329018 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:56.329676 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet e1b011e94e2b47d7a51d6816576ea124 (table default.Table_ef03c2589fde409f95b535d84b66bf29 [id=e8385d8ad6714acf91d8cb6b0de6469c]). Expected version 2 got 0
I20240506 19:56:56.330020 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]). Expected version 1 got 0
I20240506 19:56:56.330212 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:56.330507 12522 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet dcb357a7eb2147a79cc69d024bd35269 (table default.Table_86e2e3b68d354e02b29118d063456d4a [id=b6aeaf0e19d746dab4a37005dc753d65]). Expected version 2 got 1
I20240506 19:56:56.334223 13249 tablet_bootstrap.cc:492] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.334375 13249 ts_tablet_manager.cc:1367] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.004s
I20240506 19:56:56.335065 13249 raft_consensus.cc:348] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.335247 13249 raft_consensus.cc:374] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.335307 13249 raft_consensus.cc:729] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.335548 13249 consensus_queue.cc:260] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.336212 13249 ts_tablet_manager.cc:1398] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:56.341367 14557 ts_tablet_manager.cc:1888] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:56.341552 14557 log.cc:1196] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2c96f014293242be9640515697c0733b
I20240506 19:56:56.341964 14557 ts_tablet_manager.cc:1909] T 2c96f014293242be9640515697c0733b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.343621 13249 tablet_bootstrap.cc:492] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.344707 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4e97d10ae54e558920d116d8fa0ba5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:56.345335 13244 tablet_bootstrap.cc:492] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:56.345517 12921 leader_election.cc:341] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.345836 13249 tablet_bootstrap.cc:654] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.347379 13244 tablet_bootstrap.cc:654] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.349668 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4e97d10ae54e558920d116d8fa0ba5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.350060 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 936) took 2781 ms. Trace:
I20240506 19:56:56.350152 13242 rpcz_store.cc:276] 0506 19:56:53.568710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.651464 (+ 82754us) service_pool.cc:225] Handling call
0506 19:56:55.992253 (+2340789us) ts_tablet_manager.cc:1120] Deleting tablet 9e3d13322b4a4a2db3b6d425679f68ed
0506 19:56:55.992263 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.263021 (+270758us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.294459 (+ 31438us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.350039 (+ 55580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2340453}
I20240506 19:56:56.350508 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 935) took 2788 ms. Trace:
I20240506 19:56:56.350579 13242 rpcz_store.cc:276] 0506 19:56:53.562061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:53.757505 (+195444us) service_pool.cc:225] Handling call
0506 19:56:56.350421 (+2592916us) ts_tablet_manager.cc:1120] Deleting tablet 3743fc7f36a74dd9b7c01a6bfb011c9d
0506 19:56:56.350428 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.350497 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2592594}
I20240506 19:56:56.350837 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 953) took 2164 ms. Trace:
I20240506 19:56:56.350898 13242 rpcz_store.cc:276] 0506 19:56:54.186438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.213907 (+ 27469us) service_pool.cc:225] Handling call
0506 19:56:56.350767 (+2136860us) ts_tablet_manager.cc:1120] Deleting tablet edaaead0cf2e4133b8bdec62fb4770a0
0506 19:56:56.350773 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.350827 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136665}
I20240506 19:56:56.351102 13242 tablet_replica.cc:330] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.351269 13242 raft_consensus.cc:2229] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.351397 13242 raft_consensus.cc:2258] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.351868 13242 ts_tablet_manager.cc:1875] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:56.353654 12921 leader_election.cc:341] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.357477 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 9e3d13322b4a4a2db3b6d425679f68ed (table default.Table_47913566bb1846d19fddafa8a3a65bd6 [id=e3a4e004e5c64784afdcc92d35a67b5f]) successfully deleted
W20240506 19:56:56.357966 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 3743fc7f36a74dd9b7c01a6bfb011c9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3743fc7f36a74dd9b7c01a6bfb011c9d already in progress: creating tablet
W20240506 19:56:56.358404 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet edaaead0cf2e4133b8bdec62fb4770a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edaaead0cf2e4133b8bdec62fb4770a0 already in progress: creating tablet
I20240506 19:56:56.358755 14569 raft_consensus.cc:482] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.358930 14569 raft_consensus.cc:504] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.359524 14569 leader_election.cc:288] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.360599 12781 leader_election.cc:341] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 1a6e0126d9854215a0e174a76c0a0391
W20240506 19:56:56.361703 12783 leader_election.cc:341] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1a6e0126d9854215a0e174a76c0a0391
I20240506 19:56:56.361882 12783 leader_election.cc:302] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.362205 14569 raft_consensus.cc:2726] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.364513 13245 tablet_bootstrap.cc:492] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.365067 13245 ts_tablet_manager.cc:1367] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.001s
I20240506 19:56:56.366040 13245 raft_consensus.cc:348] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.366336 13245 raft_consensus.cc:374] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.366430 13245 raft_consensus.cc:729] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.366689 13245 consensus_queue.cc:260] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.371845 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 825528636a8f43188a863d2f28588a77
I20240506 19:56:56.373248 14398 raft_consensus.cc:482] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.373430 14398 raft_consensus.cc:504] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.374056 14398 leader_election.cc:288] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.374321 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:56.374532 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:56.375013 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43da1d12ce647dcbcd31263932b2f54" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.375012 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43da1d12ce647dcbcd31263932b2f54" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.375455 13242 ts_tablet_manager.cc:1888] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240506 19:56:56.375520 13057 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.375576 13242 log.cc:1196] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/378759ca310f453fba45d48879e2dd1a
W20240506 19:56:56.375720 13058 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.375797 13058 leader_election.cc:302] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.376042 14460 raft_consensus.cc:482] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.376001 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_68bc366f32d440a9a0ddab8c2bcfb05e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:56.376154 14398 raft_consensus.cc:2726] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.376168 14460 raft_consensus.cc:504] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.376562 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68bc366f32d440a9a0ddab8c2bcfb05e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.376780 14460 leader_election.cc:288] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.377355 13245 ts_tablet_manager.cc:1398] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
W20240506 19:56:56.377437 12922 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:56:56.378439 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_096d0d9806624f8dab6c61cfd67e5481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:56.378880 13033 heartbeater.cc:347] Connected to a master server at 127.12.48.252:36867
W20240506 19:56:56.378935 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_096d0d9806624f8dab6c61cfd67e5481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.378394 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f9aaca444c839a88c07f58cb6ad6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.379712 12921 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.379817 12921 leader_election.cc:302] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.380268 14460 raft_consensus.cc:2726] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.381081 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1045) took 1176 ms. Trace:
I20240506 19:56:56.381842 12519 master_service.cc:946] ReplaceTablet: received request to replace tablet c98d8aed6bc8460b9be6b8bfd00373d9 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:56.383473 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a02b198026be469282f8dc0e82181973 consensus state has no leader
I20240506 19:56:56.383697 13242 ts_tablet_manager.cc:1909] T 378759ca310f453fba45d48879e2dd1a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.383860 13245 tablet_bootstrap.cc:492] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.384971 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet cf97a98927784b7080b8f30b33fe91bf
I20240506 19:56:56.386422 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet cf97a98927784b7080b8f30b33fe91bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.386967 12521 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758] alter complete (version 2)
I20240506 19:56:56.387184 13245 tablet_bootstrap.cc:654] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.388185 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 825528636a8f43188a863d2f28588a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.383370 14557 rpcz_store.cc:276] 0506 19:56:55.204799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.217008 (+ 12209us) service_pool.cc:225] Handling call
0506 19:56:56.323044 (+1106036us) ts_tablet_manager.cc:1120] Deleting tablet 2c96f014293242be9640515697c0733b
0506 19:56:56.325691 (+ 2647us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.338189 (+ 12498us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.341255 (+ 3066us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.381061 (+ 39806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105807}
I20240506 19:56:56.389624 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 825528636a8f43188a863d2f28588a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.389947 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acde41b2c03440a89243aae008779e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.390220 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet cf97a98927784b7080b8f30b33fe91bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.392493 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 954) took 2175 ms. Trace:
I20240506 19:56:56.392885 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2c96f014293242be9640515697c0733b (table default.Table_b41fdc2ddabb43b6abd4d24fecd7447a [id=f67e8f35a154427f95ae0f771099e941]) successfully deleted
I20240506 19:56:56.389676 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 825528636a8f43188a863d2f28588a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.392664 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet cf97a98927784b7080b8f30b33fe91bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:56.394137 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet b743d0c18c9149fca7da961dade6ad80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b743d0c18c9149fca7da961dade6ad80 already in progress: creating tablet
I20240506 19:56:56.394405 13242 rpcz_store.cc:276] 0506 19:56:54.217433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.218083 (+ 650us) service_pool.cc:225] Handling call
0506 19:56:56.351066 (+2132983us) ts_tablet_manager.cc:1120] Deleting tablet 378759ca310f453fba45d48879e2dd1a
0506 19:56:56.351072 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.356556 (+ 5484us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.373879 (+ 17323us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.392475 (+ 18596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2132794}
I20240506 19:56:56.394811 13242 tablet_replica.cc:330] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.394838 12517 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]). Expected version 1 got 0
I20240506 19:56:56.394994 13242 raft_consensus.cc:2229] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.395149 13242 raft_consensus.cc:2258] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.395704 13242 ts_tablet_manager.cc:1875] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.395946 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 378759ca310f453fba45d48879e2dd1a (table default.Table_516b51665d1d453491f1c9caf37dda83 [id=95ada67289714d7d8529b5338c361fc8]) successfully deleted
W20240506 19:56:56.398420 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 904f11fe9eac4fcea9832218a4bed98b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 904f11fe9eac4fcea9832218a4bed98b already in progress: creating tablet
I20240506 19:56:56.398717 14460 raft_consensus.cc:482] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.398855 14460 raft_consensus.cc:504] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.399228 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5e0d2620e9b044d4a5393905dd527e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e0d2620e9b044d4a5393905dd527e11 already in progress: creating tablet
I20240506 19:56:56.399487 14557 tablet_replica.cc:330] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.399576 14460 leader_election.cc:288] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.399752 14557 raft_consensus.cc:2229] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.399907 14557 raft_consensus.cc:2258] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:56.400632 12922 leader_election.cc:341] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 40b9731a473449b19a9e95653a9d0830
I20240506 19:56:56.401386 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b9731a473449b19a9e95653a9d0830" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.402040 12921 leader_election.cc:341] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.402137 12921 leader_election.cc:302] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.402691 14460 raft_consensus.cc:2726] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.403316 14557 ts_tablet_manager.cc:1875] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.412309 13242 ts_tablet_manager.cc:1888] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.414216 13249 tablet_bootstrap.cc:492] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.414374 13249 ts_tablet_manager.cc:1367] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.009s
I20240506 19:56:56.415165 13249 raft_consensus.cc:348] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.415364 13249 raft_consensus.cc:374] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.415437 13249 raft_consensus.cc:729] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.415650 13249 consensus_queue.cc:260] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.416173 13249 ts_tablet_manager.cc:1398] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:56.421808 14519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:56.422049 13249 tablet_bootstrap.cc:492] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.422084 14519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:56.423730 14442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a2eabb30d4154960a44eb50a8a87d. Latest consensus state: current_term: 1 leader_uuid: "190a2eabb30d4154960a44eb50a8a87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:56.424016 14442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:56:56.427573 14519 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:56:56.429463 13249 tablet_bootstrap.cc:654] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.434486 14557 ts_tablet_manager.cc:1888] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.435456 13242 log.cc:1196] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ac9ecd2e034e49e49772db7435da3c81
I20240506 19:56:56.436249 13242 ts_tablet_manager.cc:1909] T ac9ecd2e034e49e49772db7435da3c81 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.436478 14557 log.cc:1196] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/673dc7c83f8748a09c41d51b42214aaf
I20240506 19:56:56.437258 14557 ts_tablet_manager.cc:1909] T 673dc7c83f8748a09c41d51b42214aaf P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.437652 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 955) took 2220 ms. Trace:
I20240506 19:56:56.438935 13242 rpcz_store.cc:276] 0506 19:56:54.217580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.220221 (+ 2641us) service_pool.cc:225] Handling call
0506 19:56:56.394756 (+2174535us) ts_tablet_manager.cc:1120] Deleting tablet ac9ecd2e034e49e49772db7435da3c81
0506 19:56:56.394768 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.409276 (+ 14508us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.412207 (+ 2931us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.437626 (+ 25419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174336}
I20240506 19:56:56.440007 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ac9ecd2e034e49e49772db7435da3c81 (table default.Table_9608d1af549c42e28d02df6bd141adf8 [id=feed002cdb9d4296a14ca76514e18ed4]) successfully deleted
I20240506 19:56:56.440574 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 956) took 2115 ms. Trace:
I20240506 19:56:56.440979 13249 tablet_bootstrap.cc:492] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.441121 13249 ts_tablet_manager.cc:1367] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240506 19:56:56.441115 13242 rpcz_store.cc:276] 0506 19:56:54.325185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:54.325713 (+ 528us) service_pool.cc:225] Handling call
0506 19:56:56.440429 (+2114716us) ts_tablet_manager.cc:1120] Deleting tablet 766d77bb818e429fb9b745e99adea585
0506 19:56:56.440445 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.440556 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2114400}
I20240506 19:56:56.441949 13249 raft_consensus.cc:348] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.442159 13249 raft_consensus.cc:374] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.442220 13249 raft_consensus.cc:729] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:56.442392 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 766d77bb818e429fb9b745e99adea585 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 766d77bb818e429fb9b745e99adea585 already in progress: creating tablet
I20240506 19:56:56.442445 13249 consensus_queue.cc:260] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.442946 13249 ts_tablet_manager.cc:1398] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:56.443204 13249 tablet_bootstrap.cc:492] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.443390 14557 tablet_replica.cc:330] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.444732 13249 tablet_bootstrap.cc:654] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.443238 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 987) took 1293 ms. Trace:
I20240506 19:56:56.445128 13242 rpcz_store.cc:276] 0506 19:56:55.150248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.150305 (+ 57us) service_pool.cc:225] Handling call
0506 19:56:56.443122 (+1292817us) ts_tablet_manager.cc:1120] Deleting tablet 01da81cb9ee840ed9e6640af305eaecc
0506 19:56:56.443131 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.443219 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292617}
W20240506 19:56:56.445850 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 01da81cb9ee840ed9e6640af305eaecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01da81cb9ee840ed9e6640af305eaecc already in progress: creating tablet
I20240506 19:56:56.446182 13245 tablet_bootstrap.cc:492] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.446367 13245 ts_tablet_manager.cc:1367] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20240506 19:56:56.446547 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 988) took 1296 ms. Trace:
I20240506 19:56:56.446770 13242 rpcz_store.cc:276] 0506 19:56:55.150384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.150436 (+ 52us) service_pool.cc:225] Handling call
0506 19:56:56.446214 (+1295778us) ts_tablet_manager.cc:1120] Deleting tablet 0fb9c05d703f417780e77359e19e542e
0506 19:56:56.446223 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.446529 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295653}
I20240506 19:56:56.447279 13245 raft_consensus.cc:348] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.447536 13245 raft_consensus.cc:374] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:56:56.447536 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0fb9c05d703f417780e77359e19e542e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fb9c05d703f417780e77359e19e542e already in progress: creating tablet
I20240506 19:56:56.447595 13245 raft_consensus.cc:729] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.448014 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 989) took 1277 ms. Trace:
I20240506 19:56:56.447904 13245 consensus_queue.cc:260] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.450273 13249 tablet_bootstrap.cc:492] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.450423 13249 ts_tablet_manager.cc:1367] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240506 19:56:56.451049 13249 raft_consensus.cc:348] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.451237 13249 raft_consensus.cc:374] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.451300 13249 raft_consensus.cc:729] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.451536 13249 consensus_queue.cc:260] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.452033 13249 ts_tablet_manager.cc:1398] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:56.452277 13249 tablet_bootstrap.cc:492] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.453079 13242 rpcz_store.cc:276] 0506 19:56:55.170245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.170494 (+ 249us) service_pool.cc:225] Handling call
0506 19:56:56.447915 (+1277421us) ts_tablet_manager.cc:1120] Deleting tablet 141bca5c816a494b83704a915f0edfd1
0506 19:56:56.447923 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.447999 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276941}
I20240506 19:56:56.453946 13249 tablet_bootstrap.cc:654] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.454504 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 141bca5c816a494b83704a915f0edfd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 141bca5c816a494b83704a915f0edfd1 already in progress: creating tablet
I20240506 19:56:56.455175 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 990) took 1284 ms. Trace:
I20240506 19:56:56.455621 13242 rpcz_store.cc:276] 0506 19:56:55.170423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.172142 (+ 1719us) service_pool.cc:225] Handling call
0506 19:56:56.455033 (+1282891us) ts_tablet_manager.cc:1120] Deleting tablet 182a3de861984b7d978f15a17275b6b0
0506 19:56:56.455046 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.455155 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282700}
I20240506 19:56:56.457589 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 673dc7c83f8748a09c41d51b42214aaf (table default.Table_f44e310a873e4dd28f57ab5ff4e28050 [id=59da284ab7244c98b6627a0263be2c64]) successfully deleted
W20240506 19:56:56.457921 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ec1ac39f62b4e15a267f5db2a6166c6 already in progress: creating tablet
W20240506 19:56:56.458462 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 182a3de861984b7d978f15a17275b6b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182a3de861984b7d978f15a17275b6b0 already in progress: creating tablet
I20240506 19:56:56.459843 13245 ts_tablet_manager.cc:1398] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240506 19:56:56.460193 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 991) took 1284 ms. Trace:
I20240506 19:56:56.460659 14569 raft_consensus.cc:482] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.462616 14569 raft_consensus.cc:504] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.463342 14569 leader_election.cc:288] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.465165 12783 leader_election.cc:341] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:56.465283 13245 tablet_bootstrap.cc:492] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.466598 13242 rpcz_store.cc:276] 0506 19:56:55.176178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.177018 (+ 840us) service_pool.cc:225] Handling call
0506 19:56:56.460003 (+1282985us) ts_tablet_manager.cc:1120] Deleting tablet 1980a68925be4d3da1078c78d32da866
0506 19:56:56.460020 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.460167 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282702}
I20240506 19:56:56.467051 13245 tablet_bootstrap.cc:654] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.467214 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 1980a68925be4d3da1078c78d32da866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1980a68925be4d3da1078c78d32da866 already in progress: creating tablet
I20240506 19:56:56.467761 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 992) took 1286 ms. Trace:
I20240506 19:56:56.467975 13242 rpcz_store.cc:276] 0506 19:56:55.180868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.181328 (+ 460us) service_pool.cc:225] Handling call
0506 19:56:56.467590 (+1286262us) ts_tablet_manager.cc:1120] Deleting tablet 2919538141834fb6a83259e29288acdd
0506 19:56:56.467601 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.467703 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286037}
W20240506 19:56:56.468452 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2919538141834fb6a83259e29288acdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2919538141834fb6a83259e29288acdd already in progress: creating tablet
W20240506 19:56:56.469122 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 0ec1ac39f62b4e15a267f5db2a6166c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ec1ac39f62b4e15a267f5db2a6166c6 already in progress: creating tablet
W20240506 19:56:56.469624 12781 leader_election.cc:341] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:56.469760 12781 leader_election.cc:302] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.470153 14569 raft_consensus.cc:2726] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.471560 13242 tablet_replica.cc:330] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.471975 13242 raft_consensus.cc:2229] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.472842 13242 raft_consensus.cc:2258] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.475425 13249 tablet_bootstrap.cc:492] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.475575 13249 ts_tablet_manager.cc:1367] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240506 19:56:56.476298 13249 raft_consensus.cc:348] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.476526 13249 raft_consensus.cc:374] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.476588 13249 raft_consensus.cc:729] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.476830 13242 ts_tablet_manager.cc:1875] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.476778 13249 consensus_queue.cc:260] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.477346 13249 ts_tablet_manager.cc:1398] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:56.477733 13249 tablet_bootstrap.cc:492] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:56.478994 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 825528636a8f43188a863d2f28588a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825528636a8f43188a863d2f28588a77 already in progress: creating tablet
W20240506 19:56:56.479246 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet cf97a98927784b7080b8f30b33fe91bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf97a98927784b7080b8f30b33fe91bf already in progress: creating tablet
I20240506 19:56:56.479930 13249 tablet_bootstrap.cc:654] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.480180 13244 tablet_bootstrap.cc:492] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.482662 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_b7e1ec0e7d9645cc9cf705d1ed10fd5d" } new_table_name: "default.Table_3270a7fb14b445ccbe0524f7b444e06d" modify_external_catalogs: true
I20240506 19:56:56.483552 13245 tablet_bootstrap.cc:492] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.483706 13245 ts_tablet_manager.cc:1367] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240506 19:56:56.484151 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_ab095fce093c440cb77fc15d75f7af97" } new_table_name: "default.Table_c0a9791431af47eaafcd42d67421b82e" modify_external_catalogs: true
I20240506 19:56:56.484493 13245 raft_consensus.cc:348] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.484750 13245 raft_consensus.cc:374] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.484807 13245 raft_consensus.cc:729] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.485031 13245 consensus_queue.cc:260] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.485435 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e475bad272ab485c8fa0c75f9dd69c4a" } new_table_name: "default.Table_f26ab51c669247fb9d3624a8c485509b" modify_external_catalogs: true
I20240506 19:56:56.486802 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fbd2d442ecdb4935bdff3a94b68cc073" } modify_external_catalogs: true
I20240506 19:56:56.486955 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_fbd2d442ecdb4935bdff3a94b68cc073" } modify_external_catalogs: true
I20240506 19:56:56.494099 13245 ts_tablet_manager.cc:1398] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:56.499795 13245 tablet_bootstrap.cc:492] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.501286 14557 raft_consensus.cc:2229] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.501607 14557 raft_consensus.cc:2258] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.502362 14557 ts_tablet_manager.cc:1875] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.508497 13249 tablet_bootstrap.cc:492] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.508653 13249 ts_tablet_manager.cc:1367] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.008s
I20240506 19:56:56.509436 13249 raft_consensus.cc:348] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.509639 13249 raft_consensus.cc:374] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.509678 13245 tablet_bootstrap.cc:654] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.509696 13249 raft_consensus.cc:729] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.509920 13249 consensus_queue.cc:260] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.510455 13249 ts_tablet_manager.cc:1398] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:56.510738 13249 tablet_bootstrap.cc:492] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.512063 13249 tablet_bootstrap.cc:654] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.522145 13244 ts_tablet_manager.cc:1367] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.000s
I20240506 19:56:56.522363 13242 ts_tablet_manager.cc:1888] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.522756 13242 log.cc:1196] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/7de7397db1e74b4488a3f9a70136bccb
I20240506 19:56:56.523185 13242 ts_tablet_manager.cc:1909] T 7de7397db1e74b4488a3f9a70136bccb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.523835 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 7de7397db1e74b4488a3f9a70136bccb (table default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]) successfully deleted
W20240506 19:56:56.524657 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet b743d0c18c9149fca7da961dade6ad80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b743d0c18c9149fca7da961dade6ad80 already in progress: creating tablet
W20240506 19:56:56.525323 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 904f11fe9eac4fcea9832218a4bed98b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 904f11fe9eac4fcea9832218a4bed98b already in progress: creating tablet
I20240506 19:56:56.525719 13242 tablet_replica.cc:330] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.526093 13242 raft_consensus.cc:2229] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.526382 13242 raft_consensus.cc:2258] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.527014 13242 ts_tablet_manager.cc:1875] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.528509 13244 raft_consensus.cc:348] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.537303 13244 raft_consensus.cc:374] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.539386 12844 raft_consensus.cc:3037] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.547834 13244 raft_consensus.cc:729] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.552246 14557 ts_tablet_manager.cc:1888] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.553114 13244 consensus_queue.cc:260] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.554857 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 71bfcd6f0f3c4a9597fef1461ec0efbe
I20240506 19:56:56.555160 13244 ts_tablet_manager.cc:1398] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240506 19:56:56.555418 14398 raft_consensus.cc:482] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.555548 14398 raft_consensus.cc:504] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.555754 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_76b20d55cba5431aa5b3b38c9eca1e56" } new_table_name: "default.Table_b22912910904450c99d5f1fc1be34500" modify_external_catalogs: true
I20240506 19:56:56.556170 14398 leader_election.cc:288] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.556751 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d53f3daaa2949a2b6883435f031dc37" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.557565 13058 leader_election.cc:341] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.557756 12516 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f2e21527b828489cab8c6dc854d0f813" } modify_external_catalogs: true
I20240506 19:56:56.557952 12516 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f2e21527b828489cab8c6dc854d0f813" } modify_external_catalogs: true
I20240506 19:56:56.559577 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet c98d8aed6bc8460b9be6b8bfd00373d9
I20240506 19:56:56.560014 12519 catalog_manager.cc:6554] ReplaceTablet: tablet c98d8aed6bc8460b9be6b8bfd00373d9 deleted and replaced by tablet 5b472ba7729c4f6184e07eff618407f4
I20240506 19:56:56.560377 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_6c429083621d42058b514a513fceff77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.560894 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c429083621d42058b514a513fceff77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.561013 13244 tablet_bootstrap.cc:492] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.562001 12521 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50] alter complete (version 2)
I20240506 19:56:56.562896 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_11ace0d42c9840d19108e3bfa5f1bb71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.563374 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11ace0d42c9840d19108e3bfa5f1bb71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.565352 12812 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.565613 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet c98d8aed6bc8460b9be6b8bfd00373d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b472ba7729c4f6184e07eff618407f4) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:56.567541 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 838d9e7986164ffd916d2d842dc61f6e consensus state has no leader
I20240506 19:56:56.567803 14398 raft_consensus.cc:482] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.567929 14398 raft_consensus.cc:504] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.567971 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d8deac05c2a74b5583e5b0ddfd9ad894"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:56.568500 14398 leader_election.cc:288] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.568537 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8deac05c2a74b5583e5b0ddfd9ad894 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.570658 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ecc75bab446e39c687f3c8b740361" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.571352 13058 leader_election.cc:341] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.572767 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5a3e237d14e0417db8c53a3ef2a6d4b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.573491 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a3e237d14e0417db8c53a3ef2a6d4b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.576298 13244 tablet_bootstrap.cc:654] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.558135 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d53f3daaa2949a2b6883435f031dc37" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.577720 14557 log.cc:1196] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/08dafb661b2443f5afadc6816868bac8
I20240506 19:56:56.566680 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.566696 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet c98d8aed6bc8460b9be6b8bfd00373d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b472ba7729c4f6184e07eff618407f4) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.570045 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ecc75bab446e39c687f3c8b740361" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:56.578253 13057 leader_election.cc:341] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.578383 13057 leader_election.cc:302] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:56.578959 13057 leader_election.cc:341] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.579069 13057 leader_election.cc:302] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.579373 14398 raft_consensus.cc:2726] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.579532 14398 raft_consensus.cc:2726] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.570315 13249 tablet_bootstrap.cc:492] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.579763 13249 ts_tablet_manager.cc:1367] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.008s
I20240506 19:56:56.580436 13249 raft_consensus.cc:348] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.580633 13249 raft_consensus.cc:374] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.580694 13249 raft_consensus.cc:729] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.580896 13249 consensus_queue.cc:260] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.581526 13249 ts_tablet_manager.cc:1398] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:56.581801 13249 tablet_bootstrap.cc:492] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.583143 13249 tablet_bootstrap.cc:654] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.586459 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.587952 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e04e93a5543f9934077006eaf2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.588172 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet d43da1d12ce647dcbcd31263932b2f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:51 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.588344 13096 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d53f3daaa2949a2b6883435f031dc37 with delete_type TABLET_DATA_DELETED (replaced by tablet c264c66cfec443c7a38677237ae2ba31) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.588526 13246 tablet_replica.cc:330] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:56.588725 13246 raft_consensus.cc:2229] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.588896 13246 raft_consensus.cc:2258] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.589491 13246 ts_tablet_manager.cc:1875] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.594841 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet c98d8aed6bc8460b9be6b8bfd00373d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b472ba7729c4f6184e07eff618407f4) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:56.597036 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71bfcd6f0f3c4a9597fef1461ec0efbe already in progress: creating tablet
I20240506 19:56:56.599840 13246 ts_tablet_manager.cc:1888] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.599979 13246 log.cc:1196] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/373e04e93a5543f9934077006eaf2ba9
I20240506 19:56:56.600245 13246 ts_tablet_manager.cc:1909] T 373e04e93a5543f9934077006eaf2ba9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:56.607167 14160 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54024 (request call id 0) took 7139 ms. Trace:
I20240506 19:56:56.610780 13242 ts_tablet_manager.cc:1888] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.610953 13242 log.cc:1196] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/63f56d2180b1472a8168fb3a63589065
I20240506 19:56:56.611267 13242 ts_tablet_manager.cc:1909] T 63f56d2180b1472a8168fb3a63589065 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.611752 13242 tablet_replica.cc:330] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.611953 13242 raft_consensus.cc:2229] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.612005 13246 tablet_replica.cc:330] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:56.611961 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 63f56d2180b1472a8168fb3a63589065 (table default.Table_ab69ac48aa7a4934a9f383276277cf10 [id=493f88f528114491b898c4a1156712fc]) successfully deleted
I20240506 19:56:56.612149 13242 raft_consensus.cc:2258] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.612197 13246 raft_consensus.cc:2229] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.612360 13246 raft_consensus.cc:2258] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.612697 13242 ts_tablet_manager.cc:1875] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.613785 14557 ts_tablet_manager.cc:1909] T 08dafb661b2443f5afadc6816868bac8 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.614843 14557 tablet_replica.cc:330] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.615015 13246 ts_tablet_manager.cc:1875] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.615046 14557 raft_consensus.cc:2229] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:56.615283 14557 pending_rounds.cc:62] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Trying to abort 1 pending ops.
I20240506 19:56:56.615331 14557 pending_rounds.cc:68] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024744098596519936 op_type: NO_OP noop_request { }
I20240506 19:56:56.615451 14557 raft_consensus.cc:2866] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 19:56:56.615447 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 08dafb661b2443f5afadc6816868bac8 (table default.Table_72cbb579f8c5457ab8ef64876959a3a5 [id=8c16005dc18b44a4918959ca3868f9f0]) successfully deleted
I20240506 19:56:56.615535 14557 raft_consensus.cc:2258] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:56.615759 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 796beb8e0a2c41d698bd4c05796263f1 because the tablet was not found. No further retry: Not found: Tablet not found: 796beb8e0a2c41d698bd4c05796263f1
W20240506 19:56:56.616005 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a9e90b93023e40b3bafbddf7bbcc10b2 because the tablet was not found. No further retry: Not found: Tablet not found: a9e90b93023e40b3bafbddf7bbcc10b2
I20240506 19:56:56.616031 14557 ts_tablet_manager.cc:1875] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.616802 14398 raft_consensus.cc:482] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.616953 14398 raft_consensus.cc:504] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.617632 14398 leader_election.cc:288] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.618345 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.619220 13058 leader_election.cc:341] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.619617 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 373e04e93a5543f9934077006eaf2ba9 (table default.Table_5344cd78a3dd4575a939a926d7e1eb16 [id=d6d66e91fd3540d292e12bfeac68617a]) successfully deleted
I20240506 19:56:56.636250 14383 raft_consensus.cc:482] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.636510 14383 raft_consensus.cc:504] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.637333 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 65036.5us
I20240506 19:56:56.637529 14383 leader_election.cc:288] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.638599 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20aa06290b9a4814a006c423ccd8fc46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.638741 14148 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" instance_seqno: 1715025391180391) as {username='slave'} at 127.12.48.194:43905; Asking this server to re-register.
W20240506 19:56:56.639668 12921 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.640220 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20aa06290b9a4814a006c423ccd8fc46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:56.640551 13120 raft_consensus.cc:2454] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:56.641057 12922 leader_election.cc:302] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.641551 14383 raft_consensus.cc:2781] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:56.641621 14383 raft_consensus.cc:482] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:56.641673 14383 raft_consensus.cc:3037] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.644104 14383 raft_consensus.cc:504] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.644423 13033 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:56.645270 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20aa06290b9a4814a006c423ccd8fc46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:56.645936 12921 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.647074 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20aa06290b9a4814a006c423ccd8fc46" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:56.647271 13116 raft_consensus.cc:3037] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.648301 14557 ts_tablet_manager.cc:1888] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:56.648392 14557 log.cc:1196] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/2ff3435c11854649b7daf74f3d8c33d7
I20240506 19:56:56.648718 14557 ts_tablet_manager.cc:1909] T 2ff3435c11854649b7daf74f3d8c33d7 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.648911 14148 ts_manager.cc:194] Registered new tserver with Master: 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.649971 14383 leader_election.cc:288] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.651235 13116 raft_consensus.cc:2454] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:56.651798 12922 leader_election.cc:302] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.652277 14383 raft_consensus.cc:2781] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:56.652446 14557 tablet_replica.cc:330] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.652431 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 2ff3435c11854649b7daf74f3d8c33d7 (table default.Table_6d22301588d54dcd8649bc08a1ade8f6 [id=09c0d79ce0d443079ca27266e1154dbc]) successfully deleted
I20240506 19:56:56.652565 14383 raft_consensus.cc:686] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:56.652658 14557 raft_consensus.cc:2229] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.652765 14557 raft_consensus.cc:2258] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:56.652951 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 99f4861a01404663870ce91978c8af4f because the tablet was not found. No further retry: Not found: Tablet not found: 99f4861a01404663870ce91978c8af4f
I20240506 19:56:56.653252 14557 ts_tablet_manager.cc:1875] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:56.653779 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 74854.8us
I20240506 19:56:56.655021 14617 tablet.cc:1722] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:56.661345 13246 ts_tablet_manager.cc:1888] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.666728 13245 tablet_bootstrap.cc:492] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.667343 13242 ts_tablet_manager.cc:1888] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.667502 13242 log.cc:1196] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b3e50bde3564493eb7267acb42cb8592
I20240506 19:56:56.667678 13245 ts_tablet_manager.cc:1367] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.168s user 0.005s sys 0.002s
I20240506 19:56:56.667855 13242 ts_tablet_manager.cc:1909] T b3e50bde3564493eb7267acb42cb8592 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.668522 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b3e50bde3564493eb7267acb42cb8592 (table default.Table_95be5bd2c51b4613b0152034aae6350d [id=b8f3e6ecf02f4dd299adfa1d7464e53f]) successfully deleted
I20240506 19:56:56.669263 13245 raft_consensus.cc:348] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.669567 13245 raft_consensus.cc:374] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.669659 13245 raft_consensus.cc:729] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.669921 13242 tablet_replica.cc:330] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.669941 13245 consensus_queue.cc:260] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.670141 13242 raft_consensus.cc:2229] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.670297 13242 raft_consensus.cc:2258] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.670431 13245 ts_tablet_manager.cc:1398] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:56:56.653396 14460 raft_consensus.cc:482] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.671074 14460 raft_consensus.cc:504] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.671784 14460 leader_election.cc:288] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.672731 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0d827d732e4261a7a8edb549c1c8e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.673286 12921 leader_election.cc:341] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.653129 14383 consensus_queue.cc:237] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.657454 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.675925 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0d827d732e4261a7a8edb549c1c8e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:56.676283 13120 raft_consensus.cc:2454] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:56.676934 12922 leader_election.cc:302] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.677532 14460 raft_consensus.cc:2781] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:56.677624 14460 raft_consensus.cc:482] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:56.677698 14460 raft_consensus.cc:3037] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.678802 14146 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" instance_seqno: 1715025391726585) as {username='slave'} at 127.12.48.195:52071; Asking this server to re-register.
I20240506 19:56:56.680915 14460 raft_consensus.cc:504] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.683017 13169 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:56.684554 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0d827d732e4261a7a8edb549c1c8e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:56.685088 12921 leader_election.cc:341] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.685285 14460 leader_election.cc:288] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.683782 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0d827d732e4261a7a8edb549c1c8e3" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:56.684021 13245 tablet_bootstrap.cc:492] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.686163 13120 raft_consensus.cc:3037] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.687165 13242 ts_tablet_manager.cc:1875] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.687820 13245 tablet_bootstrap.cc:654] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.689867 14656 raft_consensus.cc:482] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.689994 14656 raft_consensus.cc:504] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.690649 13057 leader_election.cc:341] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.690747 13057 leader_election.cc:302] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.665768 13249 tablet_bootstrap.cc:492] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.691082 13249 ts_tablet_manager.cc:1367] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.006s
I20240506 19:56:56.690963 14656 leader_election.cc:288] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.691650 14398 raft_consensus.cc:2726] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.691920 13246 log.cc:1196] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/d43da1d12ce647dcbcd31263932b2f54
I20240506 19:56:56.692258 13246 ts_tablet_manager.cc:1909] T d43da1d12ce647dcbcd31263932b2f54 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:56.693022 13249 raft_consensus.cc:348] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.693591 13120 raft_consensus.cc:2454] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:56.694319 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet d43da1d12ce647dcbcd31263932b2f54 (table default.Table_81e0b435a0344362af8168947f3b1d56 [id=0b3eb5aa7ebf44948d41cb63f9ea6a9f]) successfully deleted
I20240506 19:56:56.694200 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.695636 12922 leader_election.cc:302] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.696350 14146 ts_manager.cc:194] Registered new tserver with Master: 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:56:56.696346 13057 leader_election.cc:341] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.695847 13249 raft_consensus.cc:374] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.696561 13249 raft_consensus.cc:729] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.696246 14460 raft_consensus.cc:2781] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:56.696978 14460 raft_consensus.cc:686] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:56.697269 14460 consensus_queue.cc:237] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.697487 12513 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef consensus state has no leader
I20240506 19:56:56.698320 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.698832 13246 tablet_replica.cc:330] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:56.699035 13246 raft_consensus.cc:2229] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.699182 13246 raft_consensus.cc:2258] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.699028 13249 consensus_queue.cc:260] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.699712 13249 ts_tablet_manager.cc:1398] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:56.700130 13249 tablet_bootstrap.cc:492] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:56.700588 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e63baf59dc8244daad831b2987a51cfb consensus state has no leader
W20240506 19:56:56.701376 13058 leader_election.cc:341] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.701483 13058 leader_election.cc:302] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.701844 14656 raft_consensus.cc:2726] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.702193 13246 ts_tablet_manager.cc:1875] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.702533 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]). Expected version 2 got 0
I20240506 19:56:56.703253 13249 tablet_bootstrap.cc:654] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.703248 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c47cda70aad1490e8d9668e7240b0673" } modify_external_catalogs: true
I20240506 19:56:56.703420 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c47cda70aad1490e8d9668e7240b0673" } modify_external_catalogs: true
I20240506 19:56:56.703548 14557 ts_tablet_manager.cc:1888] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.703640 14557 log.cc:1196] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/85bc7cfd006f4d4ca95541dab6873d32
W20240506 19:56:56.703900 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b77db13623e34ef0aeee0b4a19eaebb2 consensus state has no leader
I20240506 19:56:56.703902 14557 ts_tablet_manager.cc:1909] T 85bc7cfd006f4d4ca95541dab6873d32 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.704173 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8b925b5a2690446181a8ccfc20c9f959" } new_table_name: "default.Table_23e6be51af4a449a8f17e9b5fed56041" modify_external_catalogs: true
I20240506 19:56:56.705490 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ad0f74c7c4cd42d293fa244de3fd693b
I20240506 19:56:56.706234 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_65858cfb30f343b5ae7629f09d0883b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.706763 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65858cfb30f343b5ae7629f09d0883b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.709092 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3eff05dacdb8400894bde49d2337d7f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.709602 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3eff05dacdb8400894bde49d2337d7f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.711094 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.711424 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.711508 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f9aaca444c839a88c07f58cb6ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.711668 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ffb88e0a0c4125b04f4d8d328da3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.711808 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.712018 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 452a83470585433e8def7174300d3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.712052 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 116df9bd6ab04a30ba09861e765d7267 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:42 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:56.714474 14460 raft_consensus.cc:482] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.714634 14460 raft_consensus.cc:504] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.715409 14460 leader_election.cc:288] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.716806 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452a83470585433e8def7174300d3a7f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.714396 13242 ts_tablet_manager.cc:1888] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.717401 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1089) took 1003 ms. Trace:
I20240506 19:56:56.717615 13242 log.cc:1196] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/bafd3c4706f74e7da36a8f3538760f43
W20240506 19:56:56.717586 12921 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:56.718263 12922 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 452a83470585433e8def7174300d3a7f
I20240506 19:56:56.718369 12922 leader_election.cc:302] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.718740 14460 raft_consensus.cc:2726] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.718637 14557 rpcz_store.cc:276] 0506 19:56:55.701150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.701559 (+ 409us) service_pool.cc:225] Handling call
0506 19:56:56.652396 (+950837us) ts_tablet_manager.cc:1120] Deleting tablet 85bc7cfd006f4d4ca95541dab6873d32
0506 19:56:56.652404 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.698143 (+ 45739us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.703431 (+ 5288us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.704825 (+ 1394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":950678}
I20240506 19:56:56.718952 14557 tablet_replica.cc:330] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.719385 13242 ts_tablet_manager.cc:1909] T bafd3c4706f74e7da36a8f3538760f43 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.719988 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 85bc7cfd006f4d4ca95541dab6873d32 (table default.Table_d909ad6f09e24952918c9471609c9d27 [id=2ebd467d8ac3476192b5634ac512aa47]) successfully deleted
I20240506 19:56:56.720362 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1021) took 1014 ms. Trace:
I20240506 19:56:56.720453 13242 rpcz_store.cc:276] 0506 19:56:55.706300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.706347 (+ 47us) service_pool.cc:225] Handling call
0506 19:56:56.669869 (+963522us) ts_tablet_manager.cc:1120] Deleting tablet bafd3c4706f74e7da36a8f3538760f43
0506 19:56:56.669878 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.691071 (+ 21193us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.714262 (+ 23191us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.720343 (+ 6081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963369}
I20240506 19:56:56.720582 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.720867 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet bafd3c4706f74e7da36a8f3538760f43 (table default.Table_162d0c1d88d04a11b0015afaea52e19f [id=5a2afe178f674b29834ccf87ec1ed218]) successfully deleted
I20240506 19:56:56.721190 14557 raft_consensus.cc:2229] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.719631 13249 tablet_bootstrap.cc:492] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.722505 13249 ts_tablet_manager.cc:1367] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.001s
I20240506 19:56:56.723742 13502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
I20240506 19:56:56.724040 14557 raft_consensus.cc:2258] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.724562 13249 raft_consensus.cc:348] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.725435 13249 raft_consensus.cc:374] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.725508 13249 raft_consensus.cc:729] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.725770 13249 consensus_queue.cc:260] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.726418 13249 ts_tablet_manager.cc:1398] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:56.726720 13249 tablet_bootstrap.cc:492] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.727130 14557 ts_tablet_manager.cc:1875] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.726742 13242 tablet_replica.cc:330] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.727588 13242 raft_consensus.cc:2229] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.729207 13242 raft_consensus.cc:2258] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.729841 14148 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" instance_seqno: 1715025390684788) as {username='slave'} at 127.12.48.193:58503; Asking this server to re-register.
I20240506 19:56:56.728082 13244 tablet_bootstrap.cc:492] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.730235 13244 ts_tablet_manager.cc:1367] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.169s user 0.016s sys 0.008s
I20240506 19:56:56.731263 13244 raft_consensus.cc:348] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.733458 13245 tablet_bootstrap.cc:492] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.733661 13245 ts_tablet_manager.cc:1367] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.003s
I20240506 19:56:56.734431 13245 raft_consensus.cc:348] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.734748 13245 raft_consensus.cc:374] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.736966 12531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:56.738548 14089 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:56:56.739529 14519 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:56.734866 13245 raft_consensus.cc:729] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.741169 13244 raft_consensus.cc:374] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.741266 13244 raft_consensus.cc:729] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.731475 12893 heartbeater.cc:464] Registering TS with master...
I20240506 19:56:56.741590 13245 consensus_queue.cc:260] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.742151 13245 ts_tablet_manager.cc:1398] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240506 19:56:56.731863 13242 ts_tablet_manager.cc:1875] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.745517 13244 consensus_queue.cc:260] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.746331 13244 ts_tablet_manager.cc:1398] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240506 19:56:56.750316 13244 tablet_bootstrap.cc:492] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.750769 13245 tablet_bootstrap.cc:492] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.751493 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0821a294cda34e98b1de703d3b5f8c1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:56.751731 13244 tablet_bootstrap.cc:654] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.752022 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0821a294cda34e98b1de703d3b5f8c1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.753396 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_1590f38624f04f70a068b387db86e4bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.753840 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1590f38624f04f70a068b387db86e4bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.754526 14656 raft_consensus.cc:482] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.754655 14656 raft_consensus.cc:504] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.754783 14557 ts_tablet_manager.cc:1888] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.754894 14557 log.cc:1196] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/d7be9788df8e47d285bd932d078c6a8b
I20240506 19:56:56.755079 12517 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9c8b5a5db9104c929597de8778ef14a1" } modify_external_catalogs: true
I20240506 19:56:56.755192 14557 ts_tablet_manager.cc:1909] T d7be9788df8e47d285bd932d078c6a8b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.755215 12517 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9c8b5a5db9104c929597de8778ef14a1" } modify_external_catalogs: true
I20240506 19:56:56.755283 14656 leader_election.cc:288] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:56.755798 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7390b6d4e5054374bed2ca2b4e4bd74f consensus state has no leader
I20240506 19:56:56.755823 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.755999 12515 master_service.cc:946] ReplaceTablet: received request to replace tablet bac098377eea4798b7a5516a2a7bb5b0 from {username='slave'} at 127.0.0.1:33592
W20240506 19:56:56.756278 13057 leader_election.cc:341] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.757442 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.757864 13242 ts_tablet_manager.cc:1888] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.757963 13242 log.cc:1196] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/03df4632adfa4b0ab999aeaa5bb4e1f4
W20240506 19:56:56.758006 13058 leader_election.cc:341] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.758100 13058 leader_election.cc:302] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.758236 13242 ts_tablet_manager.cc:1909] T 03df4632adfa4b0ab999aeaa5bb4e1f4 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.758955 14656 raft_consensus.cc:2726] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.758966 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1084) took 1059 ms. Trace:
I20240506 19:56:56.759052 14557 rpcz_store.cc:276] 0506 19:56:55.699408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.701853 (+ 2445us) service_pool.cc:225] Handling call
0506 19:56:56.718908 (+1017055us) ts_tablet_manager.cc:1120] Deleting tablet d7be9788df8e47d285bd932d078c6a8b
0506 19:56:56.718917 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.734321 (+ 15404us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.754672 (+ 20351us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.758941 (+ 4269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016902}
I20240506 19:56:56.759428 14557 tablet_replica.cc:330] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.759526 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet d7be9788df8e47d285bd932d078c6a8b (table default.Table_75c7fd329faa49bf99581532e6fd75d2 [id=8baacc24dda44dd5ad46aef596e96276]) successfully deleted
I20240506 19:56:56.759624 14557 raft_consensus.cc:2229] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.759764 14557 raft_consensus.cc:2258] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.760303 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1024) took 1053 ms. Trace:
I20240506 19:56:56.760358 13246 ts_tablet_manager.cc:1888] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.760471 13246 log.cc:1196] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1d53f3daaa2949a2b6883435f031dc37
I20240506 19:56:56.760423 13242 rpcz_store.cc:276] 0506 19:56:55.706585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.706629 (+ 44us) service_pool.cc:225] Handling call
0506 19:56:56.726683 (+1020054us) ts_tablet_manager.cc:1120] Deleting tablet 03df4632adfa4b0ab999aeaa5bb4e1f4
0506 19:56:56.726693 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.748040 (+ 21347us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.757766 (+ 9726us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.760283 (+ 2517us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019915}
I20240506 19:56:56.760751 13246 ts_tablet_manager.cc:1909] T 1d53f3daaa2949a2b6883435f031dc37 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:56.760974 13242 tablet_replica.cc:330] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.761560 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 03df4632adfa4b0ab999aeaa5bb4e1f4 (table default.Table_052649a7995c41f2a0dde09686115a96 [id=5e3d0dbf38d5451188d26847793c20a1]) successfully deleted
I20240506 19:56:56.762171 13242 raft_consensus.cc:2229] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.762338 13242 raft_consensus.cc:2258] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.762840 13242 ts_tablet_manager.cc:1875] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.765928 14557 ts_tablet_manager.cc:1875] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.769659 13246 tablet_replica.cc:330] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:56.769891 13246 raft_consensus.cc:2229] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.770057 13246 raft_consensus.cc:2258] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.772954 14148 ts_manager.cc:194] Registered new tserver with Master: ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:56.773638 12922 consensus_peers.cc:485] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:56.774227 13246 ts_tablet_manager.cc:1875] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.776950 14557 ts_tablet_manager.cc:1888] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.778386 13244 tablet_bootstrap.cc:492] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.778529 13244 ts_tablet_manager.cc:1367] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240506 19:56:56.779259 13244 raft_consensus.cc:348] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.779444 13244 raft_consensus.cc:374] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.779492 13244 raft_consensus.cc:729] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.779841 13244 consensus_queue.cc:260] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.780337 14557 log.cc:1196] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/234e4f3439e84b5fbb2f7bab1dbd0822
I20240506 19:56:56.780371 13244 ts_tablet_manager.cc:1398] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:56.780565 13242 ts_tablet_manager.cc:1888] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.780854 13242 log.cc:1196] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/9bb90bf767c94d73ace6411cfd31d65c
I20240506 19:56:56.781152 14557 ts_tablet_manager.cc:1909] T 234e4f3439e84b5fbb2f7bab1dbd0822 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.780999 13244 tablet_bootstrap.cc:492] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.782632 13244 tablet_bootstrap.cc:654] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.783984 13242 ts_tablet_manager.cc:1909] T 9bb90bf767c94d73ace6411cfd31d65c P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.785046 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1090) took 1083 ms. Trace:
I20240506 19:56:56.786553 14557 rpcz_store.cc:276] 0506 19:56:55.701990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.702083 (+ 93us) service_pool.cc:225] Handling call
0506 19:56:56.759367 (+1057284us) ts_tablet_manager.cc:1120] Deleting tablet 234e4f3439e84b5fbb2f7bab1dbd0822
0506 19:56:56.759379 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.773409 (+ 14030us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.776853 (+ 3444us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.785023 (+ 8170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057142}
I20240506 19:56:56.787371 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 234e4f3439e84b5fbb2f7bab1dbd0822 (table default.Table_4cbe3de6b961432fbeacafa9ae5fe50c [id=f6e95966d94c4b5694b843e272ea941c]) successfully deleted
I20240506 19:56:56.787935 14557 tablet_replica.cc:330] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.788378 14557 raft_consensus.cc:2229] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.788869 14557 raft_consensus.cc:2258] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.789649 14557 ts_tablet_manager.cc:1875] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.789830 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1014) took 1084 ms. Trace:
I20240506 19:56:56.789950 13242 rpcz_store.cc:276] 0506 19:56:55.705624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.706717 (+ 1093us) service_pool.cc:225] Handling call
0506 19:56:56.760792 (+1054075us) ts_tablet_manager.cc:1120] Deleting tablet 9bb90bf767c94d73ace6411cfd31d65c
0506 19:56:56.760801 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.772830 (+ 12029us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.779992 (+ 7162us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.789808 (+ 9816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053945}
I20240506 19:56:56.790477 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 9bb90bf767c94d73ace6411cfd31d65c (table default.Table_a410467878c2481f86118d12ccd06f28 [id=f339c5ce6df04d839431b2475a6c1d8d]) successfully deleted
I20240506 19:56:56.790982 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1025) took 1084 ms. Trace:
I20240506 19:56:56.791107 13242 rpcz_store.cc:276] 0506 19:56:55.706668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.706794 (+ 126us) service_pool.cc:225] Handling call
0506 19:56:56.790849 (+1084055us) ts_tablet_manager.cc:1120] Deleting tablet 673dc7c83f8748a09c41d51b42214aaf
0506 19:56:56.790861 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.790963 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083914}
W20240506 19:56:56.791636 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 673dc7c83f8748a09c41d51b42214aaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 673dc7c83f8748a09c41d51b42214aaf already in progress: creating tablet
I20240506 19:56:56.792065 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1013) took 1086 ms. Trace:
I20240506 19:56:56.792217 13242 rpcz_store.cc:276] 0506 19:56:55.705537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.705571 (+ 34us) service_pool.cc:225] Handling call
0506 19:56:56.791951 (+1086380us) ts_tablet_manager.cc:1120] Deleting tablet 5e0d2620e9b044d4a5393905dd527e11
0506 19:56:56.791961 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.792048 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084704}
W20240506 19:56:56.792678 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5e0d2620e9b044d4a5393905dd527e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e0d2620e9b044d4a5393905dd527e11 already in progress: creating tablet
I20240506 19:56:56.792872 14569 raft_consensus.cc:482] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.793044 13242 tablet_replica.cc:330] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.793020 14569 raft_consensus.cc:504] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.793244 13242 raft_consensus.cc:2229] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.793417 13242 raft_consensus.cc:2258] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.793711 14569 leader_election.cc:288] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:56.793920 13242 ts_tablet_manager.cc:1875] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.794409 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:56.796411 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5751124adb5b40f59e8d79a4605ae8c4" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:56.797138 14383 raft_consensus.cc:482] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.797338 14383 raft_consensus.cc:504] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.798167 14383 leader_election.cc:288] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.798892 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:56.799120 13120 raft_consensus.cc:2454] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:56.799669 12922 leader_election.cc:302] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:56.800422 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:56.801158 12921 leader_election.cc:341] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.801764 14383 raft_consensus.cc:2781] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:56.801869 14383 raft_consensus.cc:482] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:56.801949 14383 raft_consensus.cc:3037] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.802032 13249 tablet_bootstrap.cc:654] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.804385 14383 raft_consensus.cc:504] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.805552 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:56.805752 13120 raft_consensus.cc:3037] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:56.806951 13244 tablet_bootstrap.cc:492] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.807090 13244 ts_tablet_manager.cc:1367] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.006s
I20240506 19:56:56.807844 13244 raft_consensus.cc:348] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.808030 13244 raft_consensus.cc:374] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.808077 13244 raft_consensus.cc:729] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.808283 13244 consensus_queue.cc:260] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.808792 13244 ts_tablet_manager.cc:1398] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:56.808841 12783 leader_election.cc:302] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:56.809420 14569 raft_consensus.cc:2726] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.809906 13120 raft_consensus.cc:2454] T 10ca9a1eaffc45f486d7a7e937a3e787 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:56.810420 12922 leader_election.cc:302] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:56.810684 14460 raft_consensus.cc:2781] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:56.810809 14460 raft_consensus.cc:686] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:56.811069 14460 consensus_queue.cc:237] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.811434 14383 leader_election.cc:288] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.814661 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:56.815050 13244 tablet_bootstrap.cc:492] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:56.815321 12921 leader_election.cc:341] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.817096 13244 tablet_bootstrap.cc:654] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.818799 13246 ts_tablet_manager.cc:1888] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:56.818925 13246 log.cc:1196] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ad0f74c7c4cd42d293fa244de3fd693b
I20240506 19:56:56.820139 13246 ts_tablet_manager.cc:1909] T ad0f74c7c4cd42d293fa244de3fd693b P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:56.821616 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1d53f3daaa2949a2b6883435f031dc37 (table default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]) successfully deleted
W20240506 19:56:56.821875 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet c98d8aed6bc8460b9be6b8bfd00373d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c98d8aed6bc8460b9be6b8bfd00373d9 already in progress: creating tablet
I20240506 19:56:56.825345 13242 ts_tablet_manager.cc:1888] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.826128 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet acf5e347bb68466e811259132e3c99b1 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:56.827548 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]) successfully deleted
I20240506 19:56:56.828579 14557 ts_tablet_manager.cc:1888] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.830300 14557 log.cc:1196] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/997ecba7fbae41178a25bf7af4d152e1
I20240506 19:56:56.830960 14557 ts_tablet_manager.cc:1909] T 997ecba7fbae41178a25bf7af4d152e1 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.826772 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b59ad067c274fddbba5f71e89480dec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.831403 13242 log.cc:1196] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2f7f7171d06948388bc0fdb3514202ed
I20240506 19:56:56.832052 13242 ts_tablet_manager.cc:1909] T 2f7f7171d06948388bc0fdb3514202ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.833067 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1091) took 1130 ms. Trace:
I20240506 19:56:56.833895 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1017) took 1127 ms. Trace:
I20240506 19:56:56.835285 13242 rpcz_store.cc:276] 0506 19:56:55.705912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.707158 (+ 1246us) service_pool.cc:225] Handling call
0506 19:56:56.792993 (+1085835us) ts_tablet_manager.cc:1120] Deleting tablet 2f7f7171d06948388bc0fdb3514202ed
0506 19:56:56.793004 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.800465 (+ 7461us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.825213 (+ 24748us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.833873 (+ 8660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085709}
I20240506 19:56:56.834256 14557 rpcz_store.cc:276] 0506 19:56:55.702114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.702362 (+ 248us) service_pool.cc:225] Handling call
0506 19:56:56.787880 (+1085518us) ts_tablet_manager.cc:1120] Deleting tablet 997ecba7fbae41178a25bf7af4d152e1
0506 19:56:56.787891 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.803835 (+ 15944us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.828459 (+ 24624us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.833046 (+ 4587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085367}
I20240506 19:56:56.836103 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 997ecba7fbae41178a25bf7af4d152e1 (table default.Table_b666605912f04406bd8d879caa8c8b9b [id=85453d1f205a4fc5bcf2dd05063162f8]) successfully deleted
I20240506 19:56:56.836565 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1079) took 1176 ms. Trace:
I20240506 19:56:56.836858 13242 tablet_replica.cc:330] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.836975 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2f7f7171d06948388bc0fdb3514202ed (table default.Table_e24bec7e49144632ab0da2f0ae398f9a [id=c50645b97f4045d59c800f5df93bdf7a]) successfully deleted
I20240506 19:56:56.837338 13242 raft_consensus.cc:2229] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.837510 13242 raft_consensus.cc:2258] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.838022 13242 ts_tablet_manager.cc:1875] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.838357 14557 rpcz_store.cc:276] 0506 19:56:55.659635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.702564 (+ 42929us) service_pool.cc:225] Handling call
0506 19:56:56.836434 (+1133870us) ts_tablet_manager.cc:1120] Deleting tablet e32a40abc4e74586af79474e4b1675c5
0506 19:56:56.836444 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.836545 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133693}
W20240506 19:56:56.839143 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet e32a40abc4e74586af79474e4b1675c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e32a40abc4e74586af79474e4b1675c5 already in progress: creating tablet
I20240506 19:56:56.839637 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1092) took 1136 ms. Trace:
I20240506 19:56:56.839759 14557 rpcz_store.cc:276] 0506 19:56:55.703271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.703996 (+ 725us) service_pool.cc:225] Handling call
0506 19:56:56.839544 (+1135548us) ts_tablet_manager.cc:1120] Deleting tablet ceeae70e0d1144ea9e3e0f8727c15e6b
0506 19:56:56.839554 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.839619 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135386}
W20240506 19:56:56.840219 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ceeae70e0d1144ea9e3e0f8727c15e6b because the tablet was not found. No further retry: Not found: Tablet not found: ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:56:56.840587 14557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41490 (request call id 1082) took 1141 ms. Trace:
I20240506 19:56:56.840740 14557 rpcz_store.cc:276] 0506 19:56:55.699140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.727222 (+ 28082us) service_pool.cc:225] Handling call
0506 19:56:56.840507 (+1113285us) ts_tablet_manager.cc:1120] Deleting tablet 04738def56324a368f8182543107e32e
0506 19:56:56.840516 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.840571 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107568}
W20240506 19:56:56.841220 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 04738def56324a368f8182543107e32e because the tablet was not found. No further retry: Not found: Tablet not found: 04738def56324a368f8182543107e32e
I20240506 19:56:56.843055 12521 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:56.844085 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5a3e237d14e0417db8c53a3ef2a6d4b2" } modify_external_catalogs: true
I20240506 19:56:56.844262 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_5a3e237d14e0417db8c53a3ef2a6d4b2" } modify_external_catalogs: true
I20240506 19:56:56.845640 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d73690e3683b43168f302fd778a0492b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.845839 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4216f98cb6ac4e249ca2653518ff9e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4216f98cb6ac4e249ca2653518ff9e87 already in progress: creating tablet
W20240506 19:56:56.846144 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d73690e3683b43168f302fd778a0492b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:56.846403 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 5de5e3db2bf34d6ab23b58d552d2f800 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de5e3db2bf34d6ab23b58d552d2f800 already in progress: creating tablet
I20240506 19:56:56.847445 13244 tablet_bootstrap.cc:492] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.847575 13244 ts_tablet_manager.cc:1367] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20240506 19:56:56.847541 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_b86c2104b98d4723bfde3500950e640b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:56.847998 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b86c2104b98d4723bfde3500950e640b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:56.849406 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 1615f6167aae4dc096b9f563ab5e6215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:38 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.848300 13244 raft_consensus.cc:348] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:56.850256 13244 raft_consensus.cc:374] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.850324 13244 raft_consensus.cc:729] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.850580 13244 consensus_queue.cc:260] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:56.851161 13244 ts_tablet_manager.cc:1398] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:56.851889 13249 tablet_bootstrap.cc:492] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.851936 13244 tablet_bootstrap.cc:492] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.852028 13249 ts_tablet_manager.cc:1367] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.125s user 0.002s sys 0.002s
I20240506 19:56:56.852734 13249 raft_consensus.cc:348] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.852952 13249 raft_consensus.cc:374] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.853011 13249 raft_consensus.cc:729] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.853513 13244 tablet_bootstrap.cc:654] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.854808 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4cdb7dd4e44b45963cf93eaefe23a6 already in progress: creating tablet
W20240506 19:56:56.855657 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7ac353c6b9e541f181e20ad72775899a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ac353c6b9e541f181e20ad72775899a already in progress: creating tablet
W20240506 19:56:56.856299 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet d842127c078f4d02bca0c63531e72c0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d842127c078f4d02bca0c63531e72c0d already in progress: creating tablet
I20240506 19:56:56.856638 14557 tablet_replica.cc:330] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.856942 14557 raft_consensus.cc:2229] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.857158 14557 raft_consensus.cc:2258] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.857784 14557 ts_tablet_manager.cc:1875] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.859732 13249 consensus_queue.cc:260] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.862179 13242 ts_tablet_manager.cc:1888] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.862543 13249 ts_tablet_manager.cc:1398] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240506 19:56:56.862815 13249 tablet_bootstrap.cc:492] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.864385 13249 tablet_bootstrap.cc:654] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.867247 13242 log.cc:1196] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/034d6309de0740d0a5806b3847e334cb
I20240506 19:56:56.867828 13242 ts_tablet_manager.cc:1909] T 034d6309de0740d0a5806b3847e334cb P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.869036 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1023) took 1162 ms. Trace:
I20240506 19:56:56.870061 13242 rpcz_store.cc:276] 0506 19:56:55.706495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.706995 (+ 500us) service_pool.cc:225] Handling call
0506 19:56:56.836813 (+1129818us) ts_tablet_manager.cc:1120] Deleting tablet 034d6309de0740d0a5806b3847e334cb
0506 19:56:56.836822 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.844299 (+ 7477us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.862063 (+ 17764us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.869013 (+ 6950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1129475}
I20240506 19:56:56.870447 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c63d8995346d78633154a8f0d4c2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:37 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:56.870865 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_27ea3728f32d484faad6006e1746dad4" } new_table_name: "default.Table_fee8e613a5414e32af0d532017c321b7" modify_external_catalogs: true
I20240506 19:56:56.870970 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 034d6309de0740d0a5806b3847e334cb (table default.Table_6bc0ddcaf1a2494facb7e66847623940 [id=f464cf9965f44615a0912d7746d16922]) successfully deleted
I20240506 19:56:56.871346 13242 tablet_replica.cc:330] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.871537 13242 raft_consensus.cc:2229] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.871706 13242 raft_consensus.cc:2258] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.872233 13242 ts_tablet_manager.cc:1875] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.873255 12521 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 994710af239f409a941203bb4faade18
I20240506 19:56:56.874056 14557 ts_tablet_manager.cc:1888] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.874902 14557 log.cc:1196] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:56:56.875396 14557 ts_tablet_manager.cc:1909] T 60d1f01c0edb4f6b8a621c1643ca1afb P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.876789 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 60d1f01c0edb4f6b8a621c1643ca1afb (table default.Table_200204bb39c0432081665c45b4aa9eae [id=4884fa87524f4bdebb7c9e61f25a48a6]) successfully deleted
I20240506 19:56:56.877918 14557 tablet_replica.cc:330] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.879213 14557 raft_consensus.cc:2229] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.879590 14557 raft_consensus.cc:2258] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:56.880882 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5a4bc6e0c184237bf77ca7feac3b91b consensus state has no leader
I20240506 19:56:56.881830 14557 ts_tablet_manager.cc:1875] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.882222 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet f68f57002f8144caaaf37c2838f98e1f from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:56.888988 14460 raft_consensus.cc:482] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.889145 14460 raft_consensus.cc:504] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.889581 13242 ts_tablet_manager.cc:1888] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.889626 13249 tablet_bootstrap.cc:492] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.889689 13242 log.cc:1196] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/b899fb38bfed4e02b3708f6abec80e0a
I20240506 19:56:56.889771 13249 ts_tablet_manager.cc:1367] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20240506 19:56:56.889977 13242 ts_tablet_manager.cc:1909] T b899fb38bfed4e02b3708f6abec80e0a P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.890017 14460 leader_election.cc:288] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:56.890558 13249 raft_consensus.cc:348] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.890631 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f759b560e2ab41a6a076976ccca811d1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.890739 13249 raft_consensus.cc:374] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.890792 13249 raft_consensus.cc:729] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.890982 13249 consensus_queue.cc:260] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.891551 13249 ts_tablet_manager.cc:1398] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:56.891790 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f759b560e2ab41a6a076976ccca811d1" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:56.892310 12922 leader_election.cc:341] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:56.893751 12921 leader_election.cc:341] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:56.893859 12921 leader_election.cc:302] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:56.894030 13249 tablet_bootstrap.cc:492] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.894399 14383 raft_consensus.cc:2726] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:56.895164 14557 ts_tablet_manager.cc:1888] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.896123 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1016) took 1190 ms. Trace:
I20240506 19:56:56.896291 13242 rpcz_store.cc:276] 0506 19:56:55.705822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.707353 (+ 1531us) service_pool.cc:225] Handling call
0506 19:56:56.871290 (+1163937us) ts_tablet_manager.cc:1120] Deleting tablet b899fb38bfed4e02b3708f6abec80e0a
0506 19:56:56.871302 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.886461 (+ 15159us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.889474 (+ 3013us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.896098 (+ 6624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163805}
I20240506 19:56:56.896909 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet b899fb38bfed4e02b3708f6abec80e0a (table default.Table_d6d662a23cfb4d02be80346ea62fbe2a [id=8dca79d6b027426c8f12402280fea23a]) successfully deleted
I20240506 19:56:56.897421 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1012) took 1192 ms. Trace:
I20240506 19:56:56.897742 14557 log.cc:1196] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:56:56.897557 13242 rpcz_store.cc:276] 0506 19:56:55.705436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.705512 (+ 76us) service_pool.cc:225] Handling call
0506 19:56:56.897256 (+1191744us) ts_tablet_manager.cc:1120] Deleting tablet e32a40abc4e74586af79474e4b1675c5
0506 19:56:56.897268 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.897402 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189512}
I20240506 19:56:56.897570 13249 tablet_bootstrap.cc:654] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.899417 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet e32a40abc4e74586af79474e4b1675c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e32a40abc4e74586af79474e4b1675c5 already in progress: creating tablet
I20240506 19:56:56.899798 13242 tablet_replica.cc:330] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.900048 14557 ts_tablet_manager.cc:1909] T bd26e6b8115644cabe6532c7be1bbe95 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.900269 13242 raft_consensus.cc:2229] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.902124 13244 tablet_bootstrap.cc:492] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.902253 13244 ts_tablet_manager.cc:1367] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.005s
I20240506 19:56:56.903424 13244 raft_consensus.cc:348] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.903611 13244 raft_consensus.cc:374] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.903667 13244 raft_consensus.cc:729] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.903872 13244 consensus_queue.cc:260] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.904382 13244 ts_tablet_manager.cc:1398] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:56.904696 13242 raft_consensus.cc:2258] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.905943 14557 tablet_replica.cc:330] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.905887 13242 ts_tablet_manager.cc:1875] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.906782 14557 raft_consensus.cc:2229] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.907703 13244 tablet_bootstrap.cc:492] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.909353 13244 tablet_bootstrap.cc:654] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.911231 13245 tablet_bootstrap.cc:654] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.915243 14557 raft_consensus.cc:2258] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.916277 14639 catalog_manager.cc:1259] Loaded cluster ID: cab80095522b47ec9cab71d12c4cac98
I20240506 19:56:56.918027 12521 master_service.cc:946] ReplaceTablet: received request to replace tablet 9be3b844a9334cfabea3946844ae1af8 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:56.919009 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_8ab9abf5e45d49b1a3377c34813da069" } new_table_name: "default.Table_bd582985c60a4a99b9f5e608a1d4003a" modify_external_catalogs: true
I20240506 19:56:56.920604 14557 ts_tablet_manager.cc:1875] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.924384 13245 tablet_bootstrap.cc:492] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.901443 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet bd26e6b8115644cabe6532c7be1bbe95 (table default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5 [id=16258fb4a39c4effb6effa8e857b0b6f]) successfully deleted
I20240506 19:56:56.924870 13249 tablet_bootstrap.cc:492] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:56.925060 13249 ts_tablet_manager.cc:1367] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.004s
I20240506 19:56:56.925715 14639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: loading cluster ID for follower catalog manager: success
W20240506 19:56:56.925971 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88d16a40c7c24972861d582fb68d4da4 consensus state has no leader
I20240506 19:56:56.926012 13249 raft_consensus.cc:348] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.926245 13249 raft_consensus.cc:374] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.926313 13249 raft_consensus.cc:729] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.926410 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_edb2261b3c4944cb8218f60533811bdb" } new_table_name: "default.Table_b09bd3dfa767452bbaf2a6be079691ce" modify_external_catalogs: true
I20240506 19:56:56.929438 13245 ts_tablet_manager.cc:1367] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.179s user 0.007s sys 0.002s
I20240506 19:56:56.930223 13245 raft_consensus.cc:348] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.930748 13245 raft_consensus.cc:374] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.930809 13245 raft_consensus.cc:729] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.931192 13245 consensus_queue.cc:260] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.931737 13245 ts_tablet_manager.cc:1398] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:56.933707 13249 consensus_queue.cc:260] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:56.934064 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1d8fbd6adc24563a0c176184177e849 consensus state has no leader
I20240506 19:56:56.934479 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_730c6ad3babb49aaba5d81f48381daec" } new_table_name: "default.Table_583d4d82aebd4faaa5506048dbb7b71e" modify_external_catalogs: true
I20240506 19:56:56.939836 13249 ts_tablet_manager.cc:1398] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240506 19:56:56.940186 13249 tablet_bootstrap.cc:492] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:56.941817 13249 tablet_bootstrap.cc:654] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:56.943727 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a512993561e849239eca5c212f19af3f consensus state has no leader
I20240506 19:56:56.944114 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_643dbcd828ba4989ba329328a3708ae3" } modify_external_catalogs: true
I20240506 19:56:56.944330 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_643dbcd828ba4989ba329328a3708ae3" } modify_external_catalogs: true
I20240506 19:56:56.947501 13242 ts_tablet_manager.cc:1888] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.947680 13242 log.cc:1196] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/f8951bf4873e4382bcd42a52368fb376
I20240506 19:56:56.950156 13244 tablet_bootstrap.cc:492] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:56.950356 13244 ts_tablet_manager.cc:1367] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20240506 19:56:56.951467 13244 raft_consensus.cc:348] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.951674 13244 raft_consensus.cc:374] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:56.951745 13244 raft_consensus.cc:729] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:56.952039 13244 consensus_queue.cc:260] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.952495 14557 ts_tablet_manager.cc:1888] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.952881 13242 ts_tablet_manager.cc:1909] T f8951bf4873e4382bcd42a52368fb376 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:56.953456 14557 log.cc:1196] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:56:56.953830 14557 ts_tablet_manager.cc:1909] T da06187c83b94d03ae4ebb67a2873cf0 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:56.954006 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1019) took 1247 ms. Trace:
I20240506 19:56:56.954126 13242 rpcz_store.cc:276] 0506 19:56:55.706079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.707509 (+ 1430us) service_pool.cc:225] Handling call
0506 19:56:56.899751 (+1192242us) ts_tablet_manager.cc:1120] Deleting tablet f8951bf4873e4382bcd42a52368fb376
0506 19:56:56.899761 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.931111 (+ 31350us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.947253 (+ 16142us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.953986 (+ 6733us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191647}
I20240506 19:56:56.954556 13242 tablet_replica.cc:330] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:56.955065 13245 tablet_bootstrap.cc:492] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:56.955029 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet f8951bf4873e4382bcd42a52368fb376 (table default.Table_6974ae6398ae4559a50fb236133424b1 [id=07a119f6e5854b1090f2f47dc8c9dada]) successfully deleted
I20240506 19:56:56.955869 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet da06187c83b94d03ae4ebb67a2873cf0 (table default.Table_b261bf09f51646a0877f068d9eb2ab36 [id=034d601564f94db89efafdf7d3b35f05]) successfully deleted
I20240506 19:56:56.956291 14557 tablet_replica.cc:330] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:56.956513 14557 raft_consensus.cc:2229] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.956734 14557 raft_consensus.cc:2258] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.960145 13244 ts_tablet_manager.cc:1398] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240506 19:56:56.961040 13245 tablet_bootstrap.cc:654] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.961313 13242 raft_consensus.cc:2229] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:56.961530 13242 raft_consensus.cc:2258] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:56.963642 13242 ts_tablet_manager.cc:1875] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.965643 13244 tablet_bootstrap.cc:492] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:56.966159 14557 ts_tablet_manager.cc:1875] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:56.969413 13244 tablet_bootstrap.cc:654] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:56.983093 13242 ts_tablet_manager.cc:1888] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:56.985765 14460 raft_consensus.cc:482] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:56.985934 14460 raft_consensus.cc:504] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:56.986693 14460 leader_election.cc:288] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:56.988663 12922 leader_election.cc:341] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 31c440555cfe4195b3ae86e1f1eb6a34
I20240506 19:56:56.994091 12953 tablet_service.cc:1467] Processing CreateTablet for tablet a04e2f6627d84ccd9b69785f3f5260a7 (DEFAULT_TABLE table=default.Table_7b54e910804a4cbb8ec86b6e6f9a9246 [id=16f928ea6812485db743757c46b5348d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.994513 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04e2f6627d84ccd9b69785f3f5260a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:56.995112 12513 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:56.998474 13245 tablet_bootstrap.cc:492] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:56.998800 13245 ts_tablet_manager.cc:1367] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.003s
I20240506 19:56:56.999775 13245 raft_consensus.cc:348] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.000058 13245 raft_consensus.cc:374] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.000061 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:57.000166 13245 raft_consensus.cc:729] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.000424 13245 consensus_queue.cc:260] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.000957 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_c7551611bbef42be87218a270d30383c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:57.001230 13245 ts_tablet_manager.cc:1398] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:57.001509 13245 tablet_bootstrap.cc:492] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:57.001629 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7551611bbef42be87218a270d30383c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.003120 13245 tablet_bootstrap.cc:654] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.006394 13101 tablet_service.cc:1467] Processing CreateTablet for tablet a04e2f6627d84ccd9b69785f3f5260a7 (DEFAULT_TABLE table=default.Table_7b54e910804a4cbb8ec86b6e6f9a9246 [id=16f928ea6812485db743757c46b5348d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.006775 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04e2f6627d84ccd9b69785f3f5260a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.011020 12954 tablet_service.cc:1467] Processing CreateTablet for tablet dfec2143383a406b8fde744240d9c0de (DEFAULT_TABLE table=default.Table_e6692a9b138044fcb858cbed4ae939ed [id=a55ec5afcb4b4c2895ff47bb6252c5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.011590 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfec2143383a406b8fde744240d9c0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.013579 13096 tablet_service.cc:1467] Processing CreateTablet for tablet dfec2143383a406b8fde744240d9c0de (DEFAULT_TABLE table=default.Table_e6692a9b138044fcb858cbed4ae939ed [id=a55ec5afcb4b4c2895ff47bb6252c5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.013965 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfec2143383a406b8fde744240d9c0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.021888 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 995cf9f281d948a09a2f239eef178b8a (DEFAULT_TABLE table=default.Table_df143abf77c545eab94fa6ed375d125f [id=2c03449e752d48ceaa0e3d59d0d49dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.022292 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995cf9f281d948a09a2f239eef178b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.024313 13249 tablet_bootstrap.cc:492] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.024430 13249 ts_tablet_manager.cc:1367] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.084s user 0.006s sys 0.001s
I20240506 19:56:57.025136 13249 raft_consensus.cc:348] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.025380 13249 raft_consensus.cc:374] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.025427 13249 raft_consensus.cc:729] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.025642 13249 consensus_queue.cc:260] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.028331 13249 ts_tablet_manager.cc:1398] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:57.028625 13249 tablet_bootstrap.cc:492] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.029824 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet acf5e347bb68466e811259132e3c99b1
I20240506 19:56:57.030303 12516 catalog_manager.cc:6554] ReplaceTablet: tablet acf5e347bb68466e811259132e3c99b1 deleted and replaced by tablet e1e0cdb80cd54bcf99a5eaab85a13363
I20240506 19:56:57.031086 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet acf5e347bb68466e811259132e3c99b1 with delete_type TABLET_DATA_DELETED (replaced by tablet e1e0cdb80cd54bcf99a5eaab85a13363) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.031690 13249 tablet_bootstrap.cc:654] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.032913 12517 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 93d6e4b9f0314cf19bbdd9fd9038d97d
I20240506 19:56:57.034921 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.035085 12963 tablet_service.cc:1467] Processing CreateTablet for tablet cdcd4266c3ca480394df58f60a59dcbf (DEFAULT_TABLE table=default.Table_6b7f6251027641e399717012c63b5ac2 [id=f74234f1ff834fbd9f3e26d93cdb7665]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.035238 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 995cf9f281d948a09a2f239eef178b8a (DEFAULT_TABLE table=default.Table_df143abf77c545eab94fa6ed375d125f [id=2c03449e752d48ceaa0e3d59d0d49dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.035485 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcd4266c3ca480394df58f60a59dcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.035741 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9be3b844a9334cfabea3946844ae1af8
I20240506 19:56:57.036130 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995cf9f281d948a09a2f239eef178b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.036273 12521 catalog_manager.cc:6554] ReplaceTablet: tablet 9be3b844a9334cfabea3946844ae1af8 deleted and replaced by tablet 66da88df0ab54047b31f871015acfde9
I20240506 19:56:57.037868 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be3b844a9334cfabea3946844ae1af8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66da88df0ab54047b31f871015acfde9) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.038120 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 2e840b9c72c142f5a8049f223e89ea5a (DEFAULT_TABLE table=default.Table_1b285865923e44c3a4af1f276d67dec3 [id=cfff447d9c9d4494997e8c6b8da25906]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.038504 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e840b9c72c142f5a8049f223e89ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.038831 13085 tablet_service.cc:1467] Processing CreateTablet for tablet cdcd4266c3ca480394df58f60a59dcbf (DEFAULT_TABLE table=default.Table_6b7f6251027641e399717012c63b5ac2 [id=f74234f1ff834fbd9f3e26d93cdb7665]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.039247 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcd4266c3ca480394df58f60a59dcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.041420 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet acf5e347bb68466e811259132e3c99b1 with delete_type TABLET_DATA_DELETED (replaced by tablet e1e0cdb80cd54bcf99a5eaab85a13363) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.041680 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.042032 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 2e840b9c72c142f5a8049f223e89ea5a (DEFAULT_TABLE table=default.Table_1b285865923e44c3a4af1f276d67dec3 [id=cfff447d9c9d4494997e8c6b8da25906]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.042398 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e840b9c72c142f5a8049f223e89ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.045434 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be3b844a9334cfabea3946844ae1af8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66da88df0ab54047b31f871015acfde9) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.050522 12518 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 392e4e4fa399443cb23b5641ff4c6c39
I20240506 19:56:57.053166 13097 tablet_service.cc:1467] Processing CreateTablet for tablet ad1504fac4e6429e8be232e0263f63bf (DEFAULT_TABLE table=default.Table_0f8c273dcd61423a9055682950543663 [id=509b7c79a486427f9003363c784a5cb1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:57.053257 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet acf5e347bb68466e811259132e3c99b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acf5e347bb68466e811259132e3c99b1 already in progress: creating tablet
W20240506 19:56:57.053570 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d6e4b9f0314cf19bbdd9fd9038d97d already in progress: creating tablet
I20240506 19:56:57.053628 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1504fac4e6429e8be232e0263f63bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:57.053788 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 9be3b844a9334cfabea3946844ae1af8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9be3b844a9334cfabea3946844ae1af8 already in progress: creating tablet
I20240506 19:56:57.057575 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 6868b3aee874401aa6d7e893446988f7 (DEFAULT_TABLE table=default.Table_728d3e1c6e2f499fbbd7bc0994b341a3 [id=63fb5dfe8fe64ae199a4861c0a175152]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.057940 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6868b3aee874401aa6d7e893446988f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.061421 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 392e4e4fa399443cb23b5641ff4c6c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.061756 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 6868b3aee874401aa6d7e893446988f7 (DEFAULT_TABLE table=default.Table_728d3e1c6e2f499fbbd7bc0994b341a3 [id=63fb5dfe8fe64ae199a4861c0a175152]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.062263 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6868b3aee874401aa6d7e893446988f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.064774 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]). Expected version 3 got 0
I20240506 19:56:56.987524 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c440555cfe4195b3ae86e1f1eb6a34" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:56.993680 12816 tablet_service.cc:1467] Processing CreateTablet for tablet a04e2f6627d84ccd9b69785f3f5260a7 (DEFAULT_TABLE table=default.Table_7b54e910804a4cbb8ec86b6e6f9a9246 [id=16f928ea6812485db743757c46b5348d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:56.999804 12847 raft_consensus.cc:3037] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.066391 12822 tablet_service.cc:1467] Processing CreateTablet for tablet b156fc3b688746809597ca8bad53a3fe (DEFAULT_TABLE table=default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.066391 12814 tablet_service.cc:1467] Processing CreateTablet for tablet f70cbde638a2412cbe5bf53c85aa7d30 (DEFAULT_TABLE table=default.Table_21bad439d300445a8251e13b351a3b45 [id=69686d06e616442f9e7c5299ca9fc543]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.066578 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 565d5f6225a74c4da1aba61d30e2a8de (DEFAULT_TABLE table=default.Table_b9e1af536a204e51a898c9821976d7ac [id=4a489bb59b9545e991edc010e270af80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.066757 12813 tablet_service.cc:1467] Processing CreateTablet for tablet e8058ab228334912b82363ff4f8e4284 (DEFAULT_TABLE table=default.Table_b5b09cfa481b48eba1086b706fb28557 [id=4637f24d5dd64c3ea6e62357d9a4e686]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.067423 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 392e4e4fa399443cb23b5641ff4c6c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.067799 12960 tablet_service.cc:1467] Processing CreateTablet for tablet ad1504fac4e6429e8be232e0263f63bf (DEFAULT_TABLE table=default.Table_0f8c273dcd61423a9055682950543663 [id=509b7c79a486427f9003363c784a5cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.068163 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1504fac4e6429e8be232e0263f63bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.010793 12812 tablet_service.cc:1467] Processing CreateTablet for tablet dfec2143383a406b8fde744240d9c0de (DEFAULT_TABLE table=default.Table_e6692a9b138044fcb858cbed4ae939ed [id=a55ec5afcb4b4c2895ff47bb6252c5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.069473 12952 tablet_service.cc:1467] Processing CreateTablet for tablet f70cbde638a2412cbe5bf53c85aa7d30 (DEFAULT_TABLE table=default.Table_21bad439d300445a8251e13b351a3b45 [id=69686d06e616442f9e7c5299ca9fc543]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.069828 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70cbde638a2412cbe5bf53c85aa7d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.070901 12950 tablet_service.cc:1467] Processing CreateTablet for tablet e8058ab228334912b82363ff4f8e4284 (DEFAULT_TABLE table=default.Table_b5b09cfa481b48eba1086b706fb28557 [id=4637f24d5dd64c3ea6e62357d9a4e686]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.071209 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8058ab228334912b82363ff4f8e4284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.071220 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 565d5f6225a74c4da1aba61d30e2a8de (DEFAULT_TABLE table=default.Table_b9e1af536a204e51a898c9821976d7ac [id=4a489bb59b9545e991edc010e270af80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.071624 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565d5f6225a74c4da1aba61d30e2a8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.071722 13090 tablet_service.cc:1467] Processing CreateTablet for tablet f70cbde638a2412cbe5bf53c85aa7d30 (DEFAULT_TABLE table=default.Table_21bad439d300445a8251e13b351a3b45 [id=69686d06e616442f9e7c5299ca9fc543]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.072027 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70cbde638a2412cbe5bf53c85aa7d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.018750 14569 raft_consensus.cc:482] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.073539 12962 tablet_service.cc:1467] Processing CreateTablet for tablet b156fc3b688746809597ca8bad53a3fe (DEFAULT_TABLE table=default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.073508 14569 raft_consensus.cc:504] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.073554 13098 tablet_service.cc:1467] Processing CreateTablet for tablet e8058ab228334912b82363ff4f8e4284 (DEFAULT_TABLE table=default.Table_b5b09cfa481b48eba1086b706fb28557 [id=4637f24d5dd64c3ea6e62357d9a4e686]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.073874 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b156fc3b688746809597ca8bad53a3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.073966 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8058ab228334912b82363ff4f8e4284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.074312 14569 leader_election.cc:288] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.075299 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c7c7b2da9e403f8241a512e210c4d9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:57.075744 12921 leader_election.cc:341] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.075850 12921 leader_election.cc:302] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.076220 14460 raft_consensus.cc:2726] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.076758 14569 raft_consensus.cc:482] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.076898 14569 raft_consensus.cc:504] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.077057 13094 tablet_service.cc:1467] Processing CreateTablet for tablet b156fc3b688746809597ca8bad53a3fe (DEFAULT_TABLE table=default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.077419 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b156fc3b688746809597ca8bad53a3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.077540 14569 leader_election.cc:288] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.077576 13092 tablet_service.cc:1467] Processing CreateTablet for tablet b12a9eef38d04e5aa5c4a9784d9d740c (DEFAULT_TABLE table=default.Table_e53451477cc247e6b349f6f3be94dbd0 [id=cc6062d3d1ce4cba97ae9ef1b304dce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.076772 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5bb166d2e740f59be13faa6e92cdb3 (DEFAULT_TABLE table=default.Table_7a1c1e363dbc41dc9708971c2656dfe5 [id=3a6f58959205421cb5894d9424959ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.077896 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12a9eef38d04e5aa5c4a9784d9d740c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.078321 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:57.080354 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5bb166d2e740f59be13faa6e92cdb3 (DEFAULT_TABLE table=default.Table_7a1c1e363dbc41dc9708971c2656dfe5 [id=3a6f58959205421cb5894d9424959ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.080685 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5bb166d2e740f59be13faa6e92cdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.019475 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 995cf9f281d948a09a2f239eef178b8a (DEFAULT_TABLE table=default.Table_df143abf77c545eab94fa6ed375d125f [id=2c03449e752d48ceaa0e3d59d0d49dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.083760 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 30beced0d19f4c11bf5bd0a5d3c05af7 (DEFAULT_TABLE table=default.Table_1f73021793a44d038dfb378928d261d3 [id=c7ee5ba59ccf47c39ebbce23f5663c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.084086 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30beced0d19f4c11bf5bd0a5d3c05af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.084700 13242 log.cc:1196] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/4095f9fc28ff48f68064ec8318ff7b51
I20240506 19:56:57.085060 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5bb166d2e740f59be13faa6e92cdb3 (DEFAULT_TABLE table=default.Table_7a1c1e363dbc41dc9708971c2656dfe5 [id=3a6f58959205421cb5894d9424959ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.085412 13242 ts_tablet_manager.cc:1909] T 4095f9fc28ff48f68064ec8318ff7b51 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.085546 12966 tablet_service.cc:1467] Processing CreateTablet for tablet 30beced0d19f4c11bf5bd0a5d3c05af7 (DEFAULT_TABLE table=default.Table_1f73021793a44d038dfb378928d261d3 [id=c7ee5ba59ccf47c39ebbce23f5663c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.085919 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30beced0d19f4c11bf5bd0a5d3c05af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.086318 14557 ts_tablet_manager.cc:1888] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.086416 14557 log.cc:1196] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/a1160ab972124630a9be7abeefb4aa56
I20240506 19:56:57.086750 14557 ts_tablet_manager.cc:1909] T a1160ab972124630a9be7abeefb4aa56 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.089108 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 30beced0d19f4c11bf5bd0a5d3c05af7 (DEFAULT_TABLE table=default.Table_1f73021793a44d038dfb378928d261d3 [id=c7ee5ba59ccf47c39ebbce23f5663c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.023144 13244 tablet_bootstrap.cc:492] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.089465 13244 ts_tablet_manager.cc:1367] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.124s user 0.000s sys 0.003s
I20240506 19:56:57.089561 12819 tablet_service.cc:1467] Processing CreateTablet for tablet b12a9eef38d04e5aa5c4a9784d9d740c (DEFAULT_TABLE table=default.Table_e53451477cc247e6b349f6f3be94dbd0 [id=cc6062d3d1ce4cba97ae9ef1b304dce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.090273 13244 raft_consensus.cc:348] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:57.090492 13244 raft_consensus.cc:374] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.090543 13244 raft_consensus.cc:729] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.090772 13244 consensus_queue.cc:260] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:57.091317 13244 ts_tablet_manager.cc:1398] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:57.029950 12817 tablet_service.cc:1467] Processing CreateTablet for tablet cdcd4266c3ca480394df58f60a59dcbf (DEFAULT_TABLE table=default.Table_6b7f6251027641e399717012c63b5ac2 [id=f74234f1ff834fbd9f3e26d93cdb7665]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.032558 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet acf5e347bb68466e811259132e3c99b1 with delete_type TABLET_DATA_DELETED (replaced by tablet e1e0cdb80cd54bcf99a5eaab85a13363) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.034344 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.035068 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 2e840b9c72c142f5a8049f223e89ea5a (DEFAULT_TABLE table=default.Table_1b285865923e44c3a4af1f276d67dec3 [id=cfff447d9c9d4494997e8c6b8da25906]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.037508 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be3b844a9334cfabea3946844ae1af8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66da88df0ab54047b31f871015acfde9) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.048856 12815 tablet_service.cc:1467] Processing CreateTablet for tablet 6868b3aee874401aa6d7e893446988f7 (DEFAULT_TABLE table=default.Table_728d3e1c6e2f499fbbd7bc0994b341a3 [id=63fb5dfe8fe64ae199a4861c0a175152]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.049942 12828 tablet_service.cc:1467] Processing CreateTablet for tablet ad1504fac4e6429e8be232e0263f63bf (DEFAULT_TABLE table=default.Table_0f8c273dcd61423a9055682950543663 [id=509b7c79a486427f9003363c784a5cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.091809 12957 tablet_service.cc:1467] Processing CreateTablet for tablet b12a9eef38d04e5aa5c4a9784d9d740c (DEFAULT_TABLE table=default.Table_e53451477cc247e6b349f6f3be94dbd0 [id=cc6062d3d1ce4cba97ae9ef1b304dce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.092257 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12a9eef38d04e5aa5c4a9784d9d740c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.093837 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c7c7b2da9e403f8241a512e210c4d9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.094282 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7cc9769ad84cf6af3cfc0a45d63aa9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.052731 12826 tablet_service.cc:1514] Processing DeleteTablet for tablet 392e4e4fa399443cb23b5641ff4c6c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.099936 14460 raft_consensus.cc:482] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.100083 14460 raft_consensus.cc:504] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.100732 14460 leader_election.cc:288] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.101358 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 565d5f6225a74c4da1aba61d30e2a8de (DEFAULT_TABLE table=default.Table_b9e1af536a204e51a898c9821976d7ac [id=4a489bb59b9545e991edc010e270af80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.101796 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565d5f6225a74c4da1aba61d30e2a8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.102586 14557 tablet_replica.cc:330] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.102782 14557 raft_consensus.cc:2229] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:57.103104 14557 raft_consensus.cc:2258] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.103677 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5bb166d2e740f59be13faa6e92cdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.106065 14557 ts_tablet_manager.cc:1875] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.106839 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.107466 12921 leader_election.cc:341] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.109339 12783 leader_election.cc:302] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.109532 14460 consensus_queue.cc:1035] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:57.109843 12783 leader_election.cc:302] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.110370 14569 raft_consensus.cc:2726] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.110419 14682 raft_consensus.cc:2726] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.112290 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04e2f6627d84ccd9b69785f3f5260a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.113544 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8058ab228334912b82363ff4f8e4284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.113595 13244 tablet_bootstrap.cc:492] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.115447 13244 tablet_bootstrap.cc:654] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.103257 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.116503 13116 raft_consensus.cc:2454] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.116789 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5bb166d2e740f59be13faa6e92cdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.117200 12922 leader_election.cc:302] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.117712 14460 raft_consensus.cc:2781] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.117785 14460 raft_consensus.cc:482] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.117841 14460 raft_consensus.cc:3037] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.120218 14460 raft_consensus.cc:504] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.120851 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e840b9c72c142f5a8049f223e89ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.124521 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:57.124754 14459 raft_consensus.cc:482] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.124866 14459 raft_consensus.cc:504] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.124987 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70cbde638a2412cbe5bf53c85aa7d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.125557 14459 leader_election.cc:288] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.129570 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1504fac4e6429e8be232e0263f63bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.133515 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1020) took 1427 ms. Trace:
I20240506 19:56:57.133517 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfec2143383a406b8fde744240d9c0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.133649 13242 rpcz_store.cc:276] 0506 19:56:55.706163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.709613 (+ 3450us) service_pool.cc:225] Handling call
0506 19:56:56.954508 (+1244895us) ts_tablet_manager.cc:1120] Deleting tablet 4095f9fc28ff48f68064ec8318ff7b51
0506 19:56:56.954519 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:56.975472 (+ 20953us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:56.982976 (+ 7504us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.133492 (+150516us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6259,"tablet-delete.queue_time_us":1244682}
I20240506 19:56:57.134092 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6868b3aee874401aa6d7e893446988f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.134186 13242 tablet_replica.cc:330] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.134366 13242 raft_consensus.cc:2229] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.134497 13242 raft_consensus.cc:2258] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.134999 13242 ts_tablet_manager.cc:1875] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.137490 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565d5f6225a74c4da1aba61d30e2a8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.141649 14656 raft_consensus.cc:482] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.141705 13249 tablet_bootstrap.cc:492] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.141832 13249 ts_tablet_manager.cc:1367] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.113s user 0.010s sys 0.004s
I20240506 19:56:57.141801 14656 raft_consensus.cc:504] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.142434 14656 leader_election.cc:288] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.142832 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b156fc3b688746809597ca8bad53a3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.142784 13249 raft_consensus.cc:348] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.143028 13249 raft_consensus.cc:374] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.143088 13249 raft_consensus.cc:729] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.143311 13249 consensus_queue.cc:260] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.143815 13249 ts_tablet_manager.cc:1398] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:57.144757 13242 ts_tablet_manager.cc:1888] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.144852 13242 log.cc:1196] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/17fb091762144f6591152680a6acd29e
W20240506 19:56:57.145468 12921 leader_election.cc:341] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.145622 13244 tablet_bootstrap.cc:492] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.145754 13244 ts_tablet_manager.cc:1367] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.001s
I20240506 19:56:57.145699 14617 tablet.cc:1722] T 5751124adb5b40f59e8d79a4605ae8c4 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:57.146945 13244 raft_consensus.cc:348] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:57.147119 13244 raft_consensus.cc:374] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.147166 13244 raft_consensus.cc:729] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.147367 13244 consensus_queue.cc:260] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:57.147881 13244 ts_tablet_manager.cc:1398] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:57.148144 13244 tablet_bootstrap.cc:492] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.149791 13244 tablet_bootstrap.cc:654] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.152081 13249 tablet_bootstrap.cc:492] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.153735 13249 tablet_bootstrap.cc:654] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.153910 13244 tablet_bootstrap.cc:492] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.154031 13244 ts_tablet_manager.cc:1367] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240506 19:56:57.154470 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ff7a3638004012b08c973192c055a5" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.154769 13242 ts_tablet_manager.cc:1909] T 17fb091762144f6591152680a6acd29e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.155015 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12a9eef38d04e5aa5c4a9784d9d740c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.154774 13244 raft_consensus.cc:348] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.155436 13244 raft_consensus.cc:374] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.155499 13244 raft_consensus.cc:729] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.156116 14383 raft_consensus.cc:482] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.156177 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b803ae289344f88b604a2053881d59b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:57.156306 14383 raft_consensus.cc:504] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.155773 13244 consensus_queue.cc:260] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.156755 14557 ts_tablet_manager.cc:1888] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:57.156864 14557 log.cc:1196] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/ad0f74c7c4cd42d293fa244de3fd693b
I20240506 19:56:57.157114 13244 ts_tablet_manager.cc:1398] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:57.157516 13244 tablet_bootstrap.cc:492] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.155203 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995cf9f281d948a09a2f239eef178b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.157946 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46916e778bf34188959e1cfff084bffc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.158401 12921 leader_election.cc:341] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.158759 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30beced0d19f4c11bf5bd0a5d3c05af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.161419 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcd4266c3ca480394df58f60a59dcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.156870 14383 leader_election.cc:288] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.162892 14557 ts_tablet_manager.cc:1909] T ad0f74c7c4cd42d293fa244de3fd693b P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.165818 14557 tablet_replica.cc:330] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.166018 14557 raft_consensus.cc:2229] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.166150 14557 raft_consensus.cc:2258] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.166395 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.166592 13116 raft_consensus.cc:3037] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.166661 14557 ts_tablet_manager.cc:1875] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:57.167114 12921 leader_election.cc:341] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.168020 13057 leader_election.cc:341] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.170853 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b803ae289344f88b604a2053881d59b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.171733 13244 tablet_bootstrap.cc:654] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.155078 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1022) took 1448 ms. Trace:
I20240506 19:56:57.172621 13242 rpcz_store.cc:276] 0506 19:56:55.706406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.709854 (+ 3448us) service_pool.cc:225] Handling call
0506 19:56:57.134146 (+1424292us) ts_tablet_manager.cc:1120] Deleting tablet 17fb091762144f6591152680a6acd29e
0506 19:56:57.134154 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.138191 (+ 4037us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.144664 (+ 6473us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.155057 (+ 10393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1424179}
I20240506 19:56:57.173197 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46916e778bf34188959e1cfff084bffc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.173485 13114 raft_consensus.cc:2454] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.173816 13242 tablet_replica.cc:330] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.174000 13242 raft_consensus.cc:2229] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.174134 13242 raft_consensus.cc:2258] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.174924 13242 ts_tablet_manager.cc:1875] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.175096 14383 raft_consensus.cc:482] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.175248 14383 raft_consensus.cc:504] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.175632 13249 tablet_bootstrap.cc:492] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.175762 13249 ts_tablet_manager.cc:1367] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240506 19:56:57.176004 14383 leader_election.cc:288] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.176471 13249 raft_consensus.cc:348] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.176635 13249 raft_consensus.cc:374] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.176687 13249 raft_consensus.cc:729] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.176705 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458314e4653748e48cf646c72373fc3e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.176851 13249 consensus_queue.cc:260] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.177170 12921 leader_election.cc:341] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.177418 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 392e4e4fa399443cb23b5641ff4c6c39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 392e4e4fa399443cb23b5641ff4c6c39 already in progress: creating tablet
I20240506 19:56:57.177788 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet a1160ab972124630a9be7abeefb4aa56 (table default.Table_cd0e12d470dd429295bcb3d354afacfe [id=9bd4743be3d849cdb29c87301aece717]) successfully deleted
W20240506 19:56:57.177932 12506 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 85866.5us
I20240506 19:56:57.178210 13249 ts_tablet_manager.cc:1398] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:56:57.178486 13249 tablet_bootstrap.cc:492] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.177400 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 4095f9fc28ff48f68064ec8318ff7b51 (table default.Table_a577b6ec9dc64f0f9be2a66998f740b5 [id=5d3770cf73714f7583eb39e72f7676fc]) successfully deleted
W20240506 19:56:57.178800 12922 leader_election.cc:341] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: c1ff7a3638004012b08c973192c055a5
I20240506 19:56:57.177395 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:57.178900 12922 leader_election.cc:302] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.179096 14383 raft_consensus.cc:2726] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:57.180204 13058 leader_election.cc:341] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.180265 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458314e4653748e48cf646c72373fc3e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:57.180502 12506 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7390b6d4e5054374bed2ca2b4e4bd74f (table default.Table_23e6be51af4a449a8f17e9b5fed56041 [id=9855dc7eecda4d969b9261f88384d088], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7390b6d4e5054374bed2ca2b4e4bd74f consensus state has no leader
W20240506 19:56:57.180732 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ddae4b0d8d0943d99f54904266f293d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddae4b0d8d0943d99f54904266f293d4 already in progress: creating tablet
I20240506 19:56:57.181110 13244 tablet_bootstrap.cc:492] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.181250 13244 ts_tablet_manager.cc:1367] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
W20240506 19:56:57.181317 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 60edeed6ae3b48e2b9f5341d7e7af01d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60edeed6ae3b48e2b9f5341d7e7af01d already in progress: creating tablet
W20240506 19:56:57.181541 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet cf97a98927784b7080b8f30b33fe91bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf97a98927784b7080b8f30b33fe91bf already in progress: creating tablet
I20240506 19:56:57.181975 13244 raft_consensus.cc:348] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.182152 13244 raft_consensus.cc:374] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.182207 13244 raft_consensus.cc:729] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:57.182439 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 825528636a8f43188a863d2f28588a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825528636a8f43188a863d2f28588a77 already in progress: creating tablet
I20240506 19:56:57.182418 13244 consensus_queue.cc:260] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.182912 13244 ts_tablet_manager.cc:1398] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240506 19:56:57.183223 12508 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 107150us
I20240506 19:56:57.183284 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2bbf7e815fc44a468fb2dc706ec8ddf5" } new_table_name: "default.Table_1d9621aac59e4d35ba7cf1d1d1206f64" modify_external_catalogs: true
I20240506 19:56:57.184373 13249 tablet_bootstrap.cc:654] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.184772 13242 ts_tablet_manager.cc:1888] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.184880 13242 log.cc:1196] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/bd2c81ac92d04ddc93cc52e5c01cbc60
I20240506 19:56:57.184960 12922 leader_election.cc:302] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.185196 13242 ts_tablet_manager.cc:1909] T bd2c81ac92d04ddc93cc52e5c01cbc60 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.185554 13244 tablet_bootstrap.cc:492] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.185695 14459 raft_consensus.cc:2781] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.185772 14459 raft_consensus.cc:482] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.185840 14459 raft_consensus.cc:3037] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.188506 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a5d469d01eb3446b81546610731630c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.189131 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5d469d01eb3446b81546610731630c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.190073 12848 raft_consensus.cc:3037] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.190522 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_131e6a1d2bc240439badeffed52cd489"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:57.180322 13058 leader_election.cc:302] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.190943 13244 tablet_bootstrap.cc:654] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:57.191293 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_131e6a1d2bc240439badeffed52cd489 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.191342 14656 raft_consensus.cc:2726] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.192356 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1026) took 1444 ms. Trace:
I20240506 19:56:57.192443 13242 rpcz_store.cc:276] 0506 19:56:55.748224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.748352 (+ 128us) service_pool.cc:225] Handling call
0506 19:56:57.173762 (+1425410us) ts_tablet_manager.cc:1120] Deleting tablet bd2c81ac92d04ddc93cc52e5c01cbc60
0506 19:56:57.173774 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.178049 (+ 4275us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.184648 (+ 6599us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.192332 (+ 7684us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1424413}
I20240506 19:56:57.192898 13242 tablet_replica.cc:330] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.193079 13242 raft_consensus.cc:2229] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.193222 13242 raft_consensus.cc:2258] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.193210 12522 catalog_manager.cc:5484] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:57.193703 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (table default.Table_23747e8c1b314e4eaa4566b557389261 [id=a8b5f55150a54dcf8a19dc9e1153ea87]). Expected version 1 got 0
I20240506 19:56:57.193778 13242 ts_tablet_manager.cc:1875] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.195679 14557 ts_tablet_manager.cc:1888] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.195788 14557 log.cc:1196] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:56:57.196070 14557 ts_tablet_manager.cc:1909] T 0aa8f9aaca444c839a88c07f58cb6ad6 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
W20240506 19:56:57.196583 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71bfcd6f0f3c4a9597fef1461ec0efbe already in progress: creating tablet
I20240506 19:56:57.197333 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:57.197788 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 46916e778bf34188959e1cfff084bffc (table default.Table_fbd2d442ecdb4935bdff3a94b68cc073 [id=df889494c5f34752bf16bde1b5a80964]). Expected version 2 got 0
I20240506 19:56:57.199332 12522 catalog_manager.cc:5484] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:57.199762 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 20aa06290b9a4814a006c423ccd8fc46 (table default.Table_e050c5de47634fe69b12743c8b30d5e9 [id=75db5850c6074469bfd66e8d26699e05]). Expected version 2 got 0
I20240506 19:56:57.200321 13242 ts_tablet_manager.cc:1888] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.200418 13242 log.cc:1196] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/607b3fc2c17b4793af9b84c19066ec5e
I20240506 19:56:57.200729 13242 ts_tablet_manager.cc:1909] T 607b3fc2c17b4793af9b84c19066ec5e P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.201007 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1027) took 1439 ms. Trace:
I20240506 19:56:57.201083 13242 rpcz_store.cc:276] 0506 19:56:55.761486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.761630 (+ 144us) service_pool.cc:225] Handling call
0506 19:56:57.192849 (+1431219us) ts_tablet_manager.cc:1120] Deleting tablet 607b3fc2c17b4793af9b84c19066ec5e
0506 19:56:57.192858 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.196808 (+ 3950us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.200243 (+ 3435us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.200985 (+ 742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1431010}
I20240506 19:56:57.201696 13242 tablet_replica.cc:330] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.201871 13242 raft_consensus.cc:2229] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.202224 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 838d9e7986164ffd916d2d842dc61f6e (table default.Table_3270a7fb14b445ccbe0524f7b444e06d [id=f424bbb43b2a47449f998c21dae05cb9]). Expected version 3 got 0
I20240506 19:56:57.202720 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:57.203714 14383 raft_consensus.cc:482] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.203877 14383 raft_consensus.cc:504] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.204595 14383 leader_election.cc:288] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.204599 12522 catalog_manager.cc:5484] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:57.205207 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 9d0d827d732e4261a7a8edb549c1c8e3 (table default.Table_9eea3fa543d74708aed9bf6f5e378d89 [id=25076b7bd89a4c3bbb205acaab09690a]). Expected version 1 got 0
I20240506 19:56:57.206053 14460 leader_election.cc:288] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:57.206595 12922 leader_election.cc:341] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.206709 12922 leader_election.cc:302] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.207370 14383 raft_consensus.cc:2726] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.208029 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da091c44824bd6bb8d390a5fb01ef8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.208287 13114 raft_consensus.cc:2454] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.209466 13249 tablet_bootstrap.cc:492] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.209733 14557 tablet_replica.cc:330] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.209576 13249 ts_tablet_manager.cc:1367] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20240506 19:56:57.209928 14557 raft_consensus.cc:2229] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.210057 14557 raft_consensus.cc:2258] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.210531 14557 ts_tablet_manager.cc:1875] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.210433 13249 raft_consensus.cc:348] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.210621 13249 raft_consensus.cc:374] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.210678 13249 raft_consensus.cc:729] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.210855 13249 consensus_queue.cc:260] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.211359 13249 ts_tablet_manager.cc:1398] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:57.211607 13249 tablet_bootstrap.cc:492] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:57.212996 12507 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 44678.1us
I20240506 19:56:57.213155 13249 tablet_bootstrap.cc:654] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.213791 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 17fb091762144f6591152680a6acd29e (table default.Table_82d6bee639334fca838d2adf031efd21 [id=d8e5bdbac7204b04ba040114b49d6ee8]) successfully deleted
I20240506 19:56:57.216425 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet bd2c81ac92d04ddc93cc52e5c01cbc60 (table default.Table_ca6325447fde4d198454d0ac20e928bd [id=a5b2fb638e5a4ee2abfd610cb6274767]) successfully deleted
I20240506 19:56:57.216650 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 607b3fc2c17b4793af9b84c19066ec5e (table default.Table_c3e795b5a9724754a33f79ebe9ac2c1f [id=343720fdaa3d43b28031433246aebe9c]) successfully deleted
W20240506 19:56:57.217525 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet c98d8aed6bc8460b9be6b8bfd00373d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c98d8aed6bc8460b9be6b8bfd00373d9 already in progress: creating tablet
I20240506 19:56:57.218482 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]) successfully deleted
I20240506 19:56:57.219529 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 0aa8f9aaca444c839a88c07f58cb6ad6 (table default.Table_bef80a208c25432493da6ad772c8bf32 [id=b759e15bf3d341939e506cb620dd5448]) successfully deleted
I20240506 19:56:57.221143 14383 raft_consensus.cc:482] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.221269 14383 raft_consensus.cc:504] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.221855 14383 leader_election.cc:288] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.227669 12522 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6ace0ec43ba44060bcec2a50e1f48098 (table default.Table_61becbd043ee470ca2ee3f4969d59971 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]). Expected version 2 got 0
I20240506 19:56:57.230641 12922 leader_election.cc:302] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:57.231459 14383 raft_consensus.cc:2781] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.231545 14383 raft_consensus.cc:482] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.231619 14383 raft_consensus.cc:3037] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.234006 14557 ts_tablet_manager.cc:1888] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.234130 14557 log.cc:1196] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:56:57.234474 14557 ts_tablet_manager.cc:1909] T f8ffb88e0a0c4125b04f4d8d328da3f5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.236176 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet f8ffb88e0a0c4125b04f4d8d328da3f5 (table default.Table_e737c1fa78974f05a022649a900e2636 [id=e3446e2531334554961b65e81f9bb53c]) successfully deleted
W20240506 19:56:57.236474 12506 catalog_manager.cc:4797] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2c96f014293242be9640515697c0733b because the tablet was not found. No further retry: Not found: Tablet not found: 2c96f014293242be9640515697c0733b
I20240506 19:56:57.238868 14557 tablet_replica.cc:330] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.239202 14557 raft_consensus.cc:2229] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.239414 14557 raft_consensus.cc:2258] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.241789 14557 ts_tablet_manager.cc:1875] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.253806 13249 tablet_bootstrap.cc:492] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.253813 13244 tablet_bootstrap.cc:492] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.253971 13249 ts_tablet_manager.cc:1367] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.013s
I20240506 19:56:57.253973 13244 ts_tablet_manager.cc:1367] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.016s
I20240506 19:56:57.254717 13249 raft_consensus.cc:348] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.254909 13249 raft_consensus.cc:374] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.254952 13249 raft_consensus.cc:729] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.255126 13249 consensus_queue.cc:260] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.255714 13249 ts_tablet_manager.cc:1398] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:57.255964 13249 tablet_bootstrap.cc:492] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.256425 14383 raft_consensus.cc:504] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.257035 14383 leader_election.cc:288] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.260237 13242 raft_consensus.cc:2258] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.260910 13242 ts_tablet_manager.cc:1875] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.265538 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ace0ec43ba44060bcec2a50e1f48098" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.266496 14682 raft_consensus.cc:482] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.267277 12842 raft_consensus.cc:3037] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.266634 14682 raft_consensus.cc:504] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.268316 14682 leader_election.cc:288] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.266202 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da091c44824bd6bb8d390a5fb01ef8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:57.269887 14557 ts_tablet_manager.cc:1888] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.269994 14557 log.cc:1196] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/452a83470585433e8def7174300d3a7f
I20240506 19:56:57.270291 14557 ts_tablet_manager.cc:1909] T 452a83470585433e8def7174300d3a7f P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.271382 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 452a83470585433e8def7174300d3a7f (table default.Table_2e36b2f9be7f4af9a58215847e671f2f [id=3a1dae7697ec4d529d8bb162d1170b3a]) successfully deleted
I20240506 19:56:57.272231 13249 tablet_bootstrap.cc:654] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.272866 13244 raft_consensus.cc:348] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.273063 13244 raft_consensus.cc:374] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.273113 13244 raft_consensus.cc:729] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.273320 13244 consensus_queue.cc:260] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.273823 13244 ts_tablet_manager.cc:1398] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240506 19:56:57.274080 13244 tablet_bootstrap.cc:492] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.274955 14557 tablet_replica.cc:330] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.275157 14557 raft_consensus.cc:2229] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.275302 14557 raft_consensus.cc:2258] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.275686 13244 tablet_bootstrap.cc:654] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.275837 14557 ts_tablet_manager.cc:1875] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:57.277278 12921 leader_election.cc:341] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.277542 12921 consensus_peers.cc:485] T 6f4e97d10ae54e558920d116d8fa0ba5 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:57.277772 12921 leader_election.cc:341] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.279094 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da091c44824bd6bb8d390a5fb01ef8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.279599 12921 leader_election.cc:341] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.280587 14617 tablet.cc:1722] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:57.281440 13249 tablet_bootstrap.cc:492] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.281555 13249 ts_tablet_manager.cc:1367] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240506 19:56:57.282279 13249 raft_consensus.cc:348] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.282465 13249 raft_consensus.cc:374] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.282516 13249 raft_consensus.cc:729] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.282774 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:57.283283 13249 consensus_queue.cc:260] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.284125 13249 ts_tablet_manager.cc:1398] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:57.284546 13249 tablet_bootstrap.cc:492] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.285187 14617 tablet.cc:1722] T dcb357a7eb2147a79cc69d024bd35269 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:57.286275 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ace0ec43ba44060bcec2a50e1f48098" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.286500 13114 raft_consensus.cc:2454] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.289759 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da091c44824bd6bb8d390a5fb01ef8" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.289971 13124 raft_consensus.cc:3037] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.291960 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb5a12521044102ae55ca953f1496f9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.292013 13242 ts_tablet_manager.cc:1888] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.292151 13242 log.cc:1196] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1a6e0126d9854215a0e174a76c0a0391
I20240506 19:56:57.295044 12922 leader_election.cc:302] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.295197 13249 tablet_bootstrap.cc:654] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.295524 14460 raft_consensus.cc:2781] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.295624 14460 raft_consensus.cc:482] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.295708 14460 raft_consensus.cc:3037] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.297432 12783 leader_election.cc:302] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.297961 14682 raft_consensus.cc:2726] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.298308 14460 raft_consensus.cc:504] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.299216 14460 leader_election.cc:288] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.299726 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ace0ec43ba44060bcec2a50e1f48098" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:57.300323 12921 leader_election.cc:341] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.302124 13124 raft_consensus.cc:2454] T 45da091c44824bd6bb8d390a5fb01ef8 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.307407 12922 leader_election.cc:302] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.308194 13116 raft_consensus.cc:2454] T 838d9e7986164ffd916d2d842dc61f6e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.308490 14460 raft_consensus.cc:2781] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.308601 14460 raft_consensus.cc:686] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.308845 14460 consensus_queue.cc:237] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.313087 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bac098377eea4798b7a5516a2a7bb5b0
I20240506 19:56:57.313731 12515 catalog_manager.cc:6554] ReplaceTablet: tablet bac098377eea4798b7a5516a2a7bb5b0 deleted and replaced by tablet 3411f2ae4a4e4e1b8609292f7ac86c70
I20240506 19:56:57.314165 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f162120d90f640e09da676cac7d0c187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.314911 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f162120d90f640e09da676cac7d0c187 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.315202 14557 ts_tablet_manager.cc:1888] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.315316 14557 log.cc:1196] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/116df9bd6ab04a30ba09861e765d7267
I20240506 19:56:57.315637 14557 ts_tablet_manager.cc:1909] T 116df9bd6ab04a30ba09861e765d7267 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.316357 12513 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_86e2e3b68d354e02b29118d063456d4a [id=b6aeaf0e19d746dab4a37005dc753d65] alter complete (version 2)
I20240506 19:56:57.317113 13249 tablet_bootstrap.cc:492] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.317270 13249 ts_tablet_manager.cc:1367] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240506 19:56:57.317947 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet f476f745e0784f7f9886484367dd8c22
I20240506 19:56:57.318277 13249 raft_consensus.cc:348] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.318517 13249 raft_consensus.cc:374] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.318578 13249 raft_consensus.cc:729] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.318766 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d8de6fedf82542b3ae58875e55aca5f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:57.318876 13249 consensus_queue.cc:260] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.319420 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8de6fedf82542b3ae58875e55aca5f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.321208 13242 ts_tablet_manager.cc:1909] T 1a6e0126d9854215a0e174a76c0a0391 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.321992 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 116df9bd6ab04a30ba09861e765d7267 (table default.Table_72ce5aee893c4fd78aca451ac2b7b773 [id=db8bd7495bce4c518581981bca778450]) successfully deleted
W20240506 19:56:57.322281 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet acf5e347bb68466e811259132e3c99b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acf5e347bb68466e811259132e3c99b1 already in progress: creating tablet
W20240506 19:56:57.322491 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d6e4b9f0314cf19bbdd9fd9038d97d already in progress: creating tablet
W20240506 19:56:57.322649 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 9be3b844a9334cfabea3946844ae1af8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9be3b844a9334cfabea3946844ae1af8 already in progress: creating tablet
W20240506 19:56:57.322819 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 392e4e4fa399443cb23b5641ff4c6c39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 392e4e4fa399443cb23b5641ff4c6c39 already in progress: creating tablet
I20240506 19:56:57.325502 14702 raft_consensus.cc:482] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.325690 14702 raft_consensus.cc:504] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.325798 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet bac098377eea4798b7a5516a2a7bb5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3411f2ae4a4e4e1b8609292f7ac86c70) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.325904 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet f476f745e0784f7f9886484367dd8c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.326548 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet bac098377eea4798b7a5516a2a7bb5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3411f2ae4a4e4e1b8609292f7ac86c70) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:57.326534 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bac098377eea4798b7a5516a2a7bb5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bac098377eea4798b7a5516a2a7bb5b0 already in progress: creating tablet
I20240506 19:56:57.326786 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet f476f745e0784f7f9886484367dd8c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
W20240506 19:56:57.327920 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bac098377eea4798b7a5516a2a7bb5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bac098377eea4798b7a5516a2a7bb5b0 already in progress: creating tablet
W20240506 19:56:57.328169 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet f476f745e0784f7f9886484367dd8c22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f476f745e0784f7f9886484367dd8c22 already in progress: creating tablet
I20240506 19:56:57.332837 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9e229895c2804c09bd30a74901fd77b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.333457 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e229895c2804c09bd30a74901fd77b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:57.334960 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a152a0f6a9ed4aa5b74fe5a53688331d consensus state has no leader
I20240506 19:56:57.335472 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2da22bc9de82467f80578b673f6faab6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.336004 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2da22bc9de82467f80578b673f6faab6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.337457 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_629374150cbb4998973db994df329a42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.337929 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_629374150cbb4998973db994df329a42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:57.338021 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet f476f745e0784f7f9886484367dd8c22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f476f745e0784f7f9886484367dd8c22 already in progress: creating tablet
I20240506 19:56:57.339284 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_bd43d9d17e59466187a8090e48b210ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:57.339465 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1028) took 1568 ms. Trace:
I20240506 19:56:57.339562 13242 rpcz_store.cc:276] 0506 19:56:55.771453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.773814 (+ 2361us) service_pool.cc:225] Handling call
0506 19:56:57.201646 (+1427832us) ts_tablet_manager.cc:1120] Deleting tablet 1a6e0126d9854215a0e174a76c0a0391
0506 19:56:57.201657 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.264314 (+ 62657us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.291920 (+ 27606us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.339440 (+ 47520us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58190,"tablet-delete.queue_time_us":1427527}
W20240506 19:56:57.339781 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd43d9d17e59466187a8090e48b210ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.339936 13242 tablet_replica.cc:330] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.340035 13244 tablet_bootstrap.cc:492] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.340126 13242 raft_consensus.cc:2229] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.340240 13244 ts_tablet_manager.cc:1367] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.002s
I20240506 19:56:57.340273 13242 raft_consensus.cc:2258] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.340848 13242 ts_tablet_manager.cc:1875] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.341080 13244 raft_consensus.cc:348] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.341400 13244 raft_consensus.cc:374] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.341521 13244 raft_consensus.cc:729] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.341831 13244 consensus_queue.cc:260] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.344034 12516 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:57.345100 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a881b5a2186a46389e703fdf3f053656"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.345623 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a881b5a2186a46389e703fdf3f053656 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.345678 13244 ts_tablet_manager.cc:1398] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:56:57.346010 13244 tablet_bootstrap.cc:492] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.348199 13244 tablet_bootstrap.cc:654] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:57.351083 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 43707.1us
I20240506 19:56:57.351974 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ace0ec43ba44060bcec2a50e1f48098" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.352212 13116 raft_consensus.cc:3037] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.355913 13116 raft_consensus.cc:2454] T 6ace0ec43ba44060bcec2a50e1f48098 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.359779 13245 tablet_bootstrap.cc:492] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:57.360002 13245 ts_tablet_manager.cc:1367] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.359s user 0.006s sys 0.002s
I20240506 19:56:57.362141 14682 raft_consensus.cc:482] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.362329 14682 raft_consensus.cc:504] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.362983 14682 leader_election.cc:288] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.363845 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1a6e0126d9854215a0e174a76c0a0391 (table default.Table_826f8c31905e4a76b9bd194c717a5853 [id=577dd46fb1b44fd380574a6cd0ba2695]) successfully deleted
I20240506 19:56:57.364733 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet bac098377eea4798b7a5516a2a7bb5b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3411f2ae4a4e4e1b8609292f7ac86c70) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.368216 13249 ts_tablet_manager.cc:1398] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.051s user 0.004s sys 0.002s
I20240506 19:56:57.368379 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1b285865923e44c3a4af1f276d67dec3" } modify_external_catalogs: true
I20240506 19:56:57.368536 13249 tablet_bootstrap.cc:492] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.368721 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1b285865923e44c3a4af1f276d67dec3" } modify_external_catalogs: true
I20240506 19:56:57.370172 13249 tablet_bootstrap.cc:654] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.370357 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c96f014293242be9640515697c0733b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.373620 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet f476f745e0784f7f9886484367dd8c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.375025 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.375568 12921 leader_election.cc:341] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.378480 14702 leader_election.cc:288] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.382071 13249 tablet_bootstrap.cc:492] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.382196 13249 ts_tablet_manager.cc:1367] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240506 19:56:57.382903 13249 raft_consensus.cc:348] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.383095 13249 raft_consensus.cc:374] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.383143 13249 raft_consensus.cc:729] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.383358 13249 consensus_queue.cc:260] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.384038 13249 ts_tablet_manager.cc:1398] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:57.384370 13249 tablet_bootstrap.cc:492] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.386109 13249 tablet_bootstrap.cc:654] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.388468 13242 ts_tablet_manager.cc:1888] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.388629 13242 log.cc:1196] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/971db6ac24d54efca95695ed1c90d4ed
I20240506 19:56:57.388928 13242 ts_tablet_manager.cc:1909] T 971db6ac24d54efca95695ed1c90d4ed P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.389801 14639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: acquiring CA information for follower catalog manager: success
I20240506 19:56:57.397217 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1029) took 1509 ms. Trace:
I20240506 19:56:57.397415 13242 rpcz_store.cc:276] 0506 19:56:55.887844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.887914 (+ 70us) service_pool.cc:225] Handling call
0506 19:56:57.339891 (+1451977us) ts_tablet_manager.cc:1120] Deleting tablet 971db6ac24d54efca95695ed1c90d4ed
0506 19:56:57.339900 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.372416 (+ 32516us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.388333 (+ 15917us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.397123 (+ 8790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451694}
I20240506 19:56:57.398241 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 971db6ac24d54efca95695ed1c90d4ed (table default.Table_cc8e3aa90fea4998b3e0a4a411add544 [id=86b683f4dec545118e95b30b194e1fa5]) successfully deleted
I20240506 19:56:57.399176 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1030) took 1477 ms. Trace:
I20240506 19:56:57.399271 13242 rpcz_store.cc:276] 0506 19:56:55.921393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.921452 (+ 59us) service_pool.cc:225] Handling call
0506 19:56:57.398999 (+1477547us) ts_tablet_manager.cc:1120] Deleting tablet 4216f98cb6ac4e249ca2653518ff9e87
0506 19:56:57.399011 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.399122 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477353}
W20240506 19:56:57.399771 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4216f98cb6ac4e249ca2653518ff9e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4216f98cb6ac4e249ca2653518ff9e87 already in progress: creating tablet
I20240506 19:56:57.400655 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1031) took 1468 ms. Trace:
I20240506 19:56:57.400794 13242 rpcz_store.cc:276] 0506 19:56:55.932444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.932499 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:57.400462 (+1467963us) ts_tablet_manager.cc:1120] Deleting tablet 5de5e3db2bf34d6ab23b58d552d2f800
0506 19:56:57.400471 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.400578 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460415}
W20240506 19:56:57.401460 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 5de5e3db2bf34d6ab23b58d552d2f800 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de5e3db2bf34d6ab23b58d552d2f800 already in progress: creating tablet
I20240506 19:56:57.401993 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1032) took 1434 ms. Trace:
I20240506 19:56:57.402179 13242 rpcz_store.cc:276] 0506 19:56:55.967206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.967379 (+ 173us) service_pool.cc:225] Handling call
0506 19:56:57.401802 (+1434423us) ts_tablet_manager.cc:1120] Deleting tablet 7ac353c6b9e541f181e20ad72775899a
0506 19:56:57.401813 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.401919 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434233}
W20240506 19:56:57.402774 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7ac353c6b9e541f181e20ad72775899a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ac353c6b9e541f181e20ad72775899a already in progress: creating tablet
I20240506 19:56:57.404637 13249 tablet_bootstrap.cc:492] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.404817 13249 ts_tablet_manager.cc:1367] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20240506 19:56:57.405849 13249 raft_consensus.cc:348] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.406167 13249 raft_consensus.cc:374] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.406289 13249 raft_consensus.cc:729] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.406598 13249 consensus_queue.cc:260] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.407331 13249 ts_tablet_manager.cc:1398] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:57.407649 13249 tablet_bootstrap.cc:492] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.407951 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1033) took 1439 ms. Trace:
I20240506 19:56:57.409577 13249 tablet_bootstrap.cc:654] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.409754 13242 rpcz_store.cc:276] 0506 19:56:55.968442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:55.968487 (+ 45us) service_pool.cc:225] Handling call
0506 19:56:57.407801 (+1439314us) ts_tablet_manager.cc:1120] Deleting tablet ac4cdb7dd4e44b45963cf93eaefe23a6
0506 19:56:57.407810 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.407897 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439161}
W20240506 19:56:57.410447 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ac4cdb7dd4e44b45963cf93eaefe23a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4cdb7dd4e44b45963cf93eaefe23a6 already in progress: creating tablet
I20240506 19:56:57.411116 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1039) took 1330 ms. Trace:
I20240506 19:56:57.411298 13242 rpcz_store.cc:276] 0506 19:56:56.080428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.080814 (+ 386us) service_pool.cc:225] Handling call
0506 19:56:57.410910 (+1330096us) ts_tablet_manager.cc:1120] Deleting tablet d842127c078f4d02bca0c63531e72c0d
0506 19:56:57.410923 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.411043 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329875}
W20240506 19:56:57.411893 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet d842127c078f4d02bca0c63531e72c0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d842127c078f4d02bca0c63531e72c0d already in progress: creating tablet
I20240506 19:56:57.412513 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1046) took 1147 ms. Trace:
I20240506 19:56:57.412650 13242 rpcz_store.cc:276] 0506 19:56:56.264771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.264847 (+ 76us) service_pool.cc:225] Handling call
0506 19:56:57.412375 (+1147528us) ts_tablet_manager.cc:1120] Deleting tablet ddae4b0d8d0943d99f54904266f293d4
0506 19:56:57.412385 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.412462 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147204}
W20240506 19:56:57.413084 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ddae4b0d8d0943d99f54904266f293d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddae4b0d8d0943d99f54904266f293d4 already in progress: creating tablet
I20240506 19:56:57.413497 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1047) took 1148 ms. Trace:
I20240506 19:56:57.416110 12922 leader_election.cc:302] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.416522 12922 leader_election.cc:302] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.416921 14700 raft_consensus.cc:2781] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.417042 14700 raft_consensus.cc:686] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.417343 14700 consensus_queue.cc:237] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.422540 14617 tablet.cc:1722] T 4e7cc9769ad84cf6af3cfc0a45d63aa9 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:57.429661 14697 raft_consensus.cc:482] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.429823 14697 raft_consensus.cc:504] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.430480 14697 leader_election.cc:288] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.431334 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a8531a1793404699532295164a3895" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.431913 12921 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.437346 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 646fa0d9bd7140be95eef34839fe8acb (DEFAULT_TABLE table=default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5 [id=7a0f775e3b564c95bd5b8a1139facf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.437666 12950 tablet_service.cc:1467] Processing CreateTablet for tablet 70faf0c82cc84bc7befdfdea22be8a15 (DEFAULT_TABLE table=default.Table_d630c009c25b457388c1041305861480 [id=0c5aabe04b194b59b5b6adcd7389744a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.437824 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646fa0d9bd7140be95eef34839fe8acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.438023 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70faf0c82cc84bc7befdfdea22be8a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.438155 12814 tablet_service.cc:1467] Processing CreateTablet for tablet 70faf0c82cc84bc7befdfdea22be8a15 (DEFAULT_TABLE table=default.Table_d630c009c25b457388c1041305861480 [id=0c5aabe04b194b59b5b6adcd7389744a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.438513 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70faf0c82cc84bc7befdfdea22be8a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.441632 12818 tablet_service.cc:1467] Processing CreateTablet for tablet a5ce47bbe8ed41488d3edded76c49978 (DEFAULT_TABLE table=default.Table_2094814385574142ae32d1a198f4b70b [id=6bf661581d744000bd9a200381c26fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.441982 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ce47bbe8ed41488d3edded76c49978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.444669 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 646fa0d9bd7140be95eef34839fe8acb (DEFAULT_TABLE table=default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5 [id=7a0f775e3b564c95bd5b8a1139facf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.445439 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646fa0d9bd7140be95eef34839fe8acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.447069 14682 raft_consensus.cc:482] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.447217 14682 raft_consensus.cc:504] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.447907 14682 leader_election.cc:288] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.456928 12950 tablet_service.cc:1467] Processing CreateTablet for tablet a5ce47bbe8ed41488d3edded76c49978 (DEFAULT_TABLE table=default.Table_2094814385574142ae32d1a198f4b70b [id=6bf661581d744000bd9a200381c26fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.459653 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ce47bbe8ed41488d3edded76c49978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.465374 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 646fa0d9bd7140be95eef34839fe8acb (DEFAULT_TABLE table=default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5 [id=7a0f775e3b564c95bd5b8a1139facf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.465916 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646fa0d9bd7140be95eef34839fe8acb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:57.466926 12783 leader_election.cc:341] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:57.466974 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 70faf0c82cc84bc7befdfdea22be8a15 (DEFAULT_TABLE table=default.Table_d630c009c25b457388c1041305861480 [id=0c5aabe04b194b59b5b6adcd7389744a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.467311 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70faf0c82cc84bc7befdfdea22be8a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.474293 13249 tablet_bootstrap.cc:492] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.474323 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]). Expected version 2 got 0
I20240506 19:56:57.474501 13249 ts_tablet_manager.cc:1367] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.002s
I20240506 19:56:57.474632 12514 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]). Expected version 2 got 0
I20240506 19:56:57.475191 13249 raft_consensus.cc:348] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.475359 13249 raft_consensus.cc:374] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.475399 13249 raft_consensus.cc:729] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.475587 13249 consensus_queue.cc:260] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.475680 12518 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_c09c5cd32c53422ca015e46ffcb2cfe1" } new_table_name: "default.Table_df06dbccf0ce480489e9ff86ef1be62d" modify_external_catalogs: true
I20240506 19:56:57.476150 13249 ts_tablet_manager.cc:1398] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:57.476378 13249 tablet_bootstrap.cc:492] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.477221 12517 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_a0ca5a4363dd40c6840fe7989c885cb0" } new_table_name: "default.Table_eaf5affd39594005b7d132e27a69442a" modify_external_catalogs: true
I20240506 19:56:57.478590 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d12f952acabc497c9f8f644888a0a361" } new_table_name: "default.Table_f659f7ad515e4001891b15f486dc7b1a" modify_external_catalogs: true
W20240506 19:56:57.480872 12783 leader_election.cc:341] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2c96f014293242be9640515697c0733b
I20240506 19:56:57.481415 13085 tablet_service.cc:1467] Processing CreateTablet for tablet a5ce47bbe8ed41488d3edded76c49978 (DEFAULT_TABLE table=default.Table_2094814385574142ae32d1a198f4b70b [id=6bf661581d744000bd9a200381c26fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.483289 13249 tablet_bootstrap.cc:654] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.487197 12818 tablet_service.cc:1467] Processing CreateTablet for tablet e77a0adbb32b479199693550f4bd0e27 (DEFAULT_TABLE table=default.Table_0c23ca41c31d456e89fdd0046e87d6f9 [id=eac26d874c4747b0b7eb9f353ccd32a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.487762 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77a0adbb32b479199693550f4bd0e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.493950 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ce47bbe8ed41488d3edded76c49978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.500586 13249 tablet_bootstrap.cc:492] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.500757 13249 ts_tablet_manager.cc:1367] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240506 19:56:57.501649 13249 raft_consensus.cc:348] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.501864 13249 raft_consensus.cc:374] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.501927 13249 raft_consensus.cc:729] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.502151 13249 consensus_queue.cc:260] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.502699 13249 ts_tablet_manager.cc:1398] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:57.502997 13249 tablet_bootstrap.cc:492] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.504494 13249 tablet_bootstrap.cc:654] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.506762 13242 rpcz_store.cc:276] 0506 19:56:56.264916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.265216 (+ 300us) service_pool.cc:225] Handling call
0506 19:56:57.413368 (+1148152us) ts_tablet_manager.cc:1120] Deleting tablet 60edeed6ae3b48e2b9f5341d7e7af01d
0506 19:56:57.413376 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.413448 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147905}
W20240506 19:56:57.507413 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 60edeed6ae3b48e2b9f5341d7e7af01d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60edeed6ae3b48e2b9f5341d7e7af01d already in progress: creating tablet
I20240506 19:56:57.507974 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1048) took 1118 ms. Trace:
I20240506 19:56:57.508131 13242 rpcz_store.cc:276] 0506 19:56:56.389514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.389569 (+ 55us) service_pool.cc:225] Handling call
0506 19:56:57.507829 (+1118260us) ts_tablet_manager.cc:1120] Deleting tablet 825528636a8f43188a863d2f28588a77
0506 19:56:57.507840 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.507953 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115919}
W20240506 19:56:57.508639 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 825528636a8f43188a863d2f28588a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825528636a8f43188a863d2f28588a77 already in progress: creating tablet
I20240506 19:56:57.508967 13242 tablet_replica.cc:330] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.510820 13242 raft_consensus.cc:2229] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.511075 13242 raft_consensus.cc:2258] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.512187 12957 tablet_service.cc:1467] Processing CreateTablet for tablet e77a0adbb32b479199693550f4bd0e27 (DEFAULT_TABLE table=default.Table_0c23ca41c31d456e89fdd0046e87d6f9 [id=eac26d874c4747b0b7eb9f353ccd32a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.512172 13242 ts_tablet_manager.cc:1875] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.512634 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77a0adbb32b479199693550f4bd0e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.513671 12809 tablet_service.cc:1467] Processing CreateTablet for tablet de82135b9e3745808395a881cf67e539 (DEFAULT_TABLE table=default.Table_9b101aa14e0a4b1cbb4aefcc7dc75d2d [id=33729edfc83a4cb19d3594b2ce4c606d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.514030 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de82135b9e3745808395a881cf67e539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.516499 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 858d58e1b2d64f47a097bb5eb7a236c7 (DEFAULT_TABLE table=default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.516558 12961 tablet_service.cc:1467] Processing CreateTablet for tablet ebecd797899b41e2b4b00e0485922add (DEFAULT_TABLE table=default.Table_aa81775ed7b04f91a8a7b6a3619fed4c [id=4f6c511cf5d04a3dba1b46ae7177a533]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.516851 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858d58e1b2d64f47a097bb5eb7a236c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.517063 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebecd797899b41e2b4b00e0485922add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.517560 12816 tablet_service.cc:1467] Processing CreateTablet for tablet ebecd797899b41e2b4b00e0485922add (DEFAULT_TABLE table=default.Table_aa81775ed7b04f91a8a7b6a3619fed4c [id=4f6c511cf5d04a3dba1b46ae7177a533]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.517885 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebecd797899b41e2b4b00e0485922add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.523895 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 858d58e1b2d64f47a097bb5eb7a236c7 (DEFAULT_TABLE table=default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.524307 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858d58e1b2d64f47a097bb5eb7a236c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.530829 13249 tablet_bootstrap.cc:492] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.531071 13249 ts_tablet_manager.cc:1367] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20240506 19:56:57.532356 13249 raft_consensus.cc:348] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.532586 13242 ts_tablet_manager.cc:1888] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.532707 13242 log.cc:1196] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/4acde41b2c03440a89243aae008779e1
I20240506 19:56:57.532711 13249 raft_consensus.cc:374] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.532979 13242 ts_tablet_manager.cc:1909] T 4acde41b2c03440a89243aae008779e1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.532966 13249 raft_consensus.cc:729] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.533850 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1049) took 1144 ms. Trace:
I20240506 19:56:57.533953 13242 rpcz_store.cc:276] 0506 19:56:56.389814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.389910 (+ 96us) service_pool.cc:225] Handling call
0506 19:56:57.508922 (+1119012us) ts_tablet_manager.cc:1120] Deleting tablet 4acde41b2c03440a89243aae008779e1
0506 19:56:57.508932 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.529256 (+ 20324us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.532492 (+ 3236us) tablet_metadata.cc:641] Metadata flushed
0506 19:56:57.533826 (+ 1334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116925}
I20240506 19:56:57.534380 13249 consensus_queue.cc:260] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.534475 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 4acde41b2c03440a89243aae008779e1 (table default.Table_fe3e87d2202147518431c995667b07fc [id=b7788b3408254775a7b73c81911108c2]) successfully deleted
I20240506 19:56:57.534942 13249 ts_tablet_manager.cc:1398] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:56:57.535207 13249 tablet_bootstrap.cc:492] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.536674 14682 raft_consensus.cc:482] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.536752 13249 tablet_bootstrap.cc:654] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.536821 14682 raft_consensus.cc:504] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.537514 14682 leader_election.cc:288] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.537906 12950 tablet_service.cc:1467] Processing CreateTablet for tablet de82135b9e3745808395a881cf67e539 (DEFAULT_TABLE table=default.Table_9b101aa14e0a4b1cbb4aefcc7dc75d2d [id=33729edfc83a4cb19d3594b2ce4c606d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.538269 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de82135b9e3745808395a881cf67e539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.541041 14459 raft_consensus.cc:504] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.542685 13242 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50504 (request call id 1050) took 1152 ms. Trace:
I20240506 19:56:57.542783 13242 rpcz_store.cc:276] 0506 19:56:56.390126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.390175 (+ 49us) service_pool.cc:225] Handling call
0506 19:56:57.542557 (+1152382us) ts_tablet_manager.cc:1120] Deleting tablet cf97a98927784b7080b8f30b33fe91bf
0506 19:56:57.542568 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0506 19:56:57.542666 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150509}
W20240506 19:56:57.543406 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet cf97a98927784b7080b8f30b33fe91bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf97a98927784b7080b8f30b33fe91bf already in progress: creating tablet
I20240506 19:56:57.546274 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:57.547083 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 71bfcd6f0f3c4a9597fef1461ec0efbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71bfcd6f0f3c4a9597fef1461ec0efbe already in progress: creating tablet
W20240506 19:56:57.547890 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet c98d8aed6bc8460b9be6b8bfd00373d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c98d8aed6bc8460b9be6b8bfd00373d9 already in progress: creating tablet
I20240506 19:56:57.548732 13088 tablet_service.cc:1467] Processing CreateTablet for tablet e77a0adbb32b479199693550f4bd0e27 (DEFAULT_TABLE table=default.Table_0c23ca41c31d456e89fdd0046e87d6f9 [id=eac26d874c4747b0b7eb9f353ccd32a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.548918 13091 tablet_service.cc:1467] Processing CreateTablet for tablet de82135b9e3745808395a881cf67e539 (DEFAULT_TABLE table=default.Table_9b101aa14e0a4b1cbb4aefcc7dc75d2d [id=33729edfc83a4cb19d3594b2ce4c606d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.549373 13085 tablet_service.cc:1467] Processing CreateTablet for tablet ebecd797899b41e2b4b00e0485922add (DEFAULT_TABLE table=default.Table_aa81775ed7b04f91a8a7b6a3619fed4c [id=4f6c511cf5d04a3dba1b46ae7177a533]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:57.553911 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ad0f74c7c4cd42d293fa244de3fd693b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad0f74c7c4cd42d293fa244de3fd693b already in progress: creating tablet
I20240506 19:56:57.558588 13090 tablet_service.cc:1467] Processing CreateTablet for tablet 858d58e1b2d64f47a097bb5eb7a236c7 (DEFAULT_TABLE table=default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:57.561525 13242 tablet_replica.cc:330] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.561764 13242 raft_consensus.cc:2229] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.561950 13242 raft_consensus.cc:2258] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.562551 13242 ts_tablet_manager.cc:1875] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.564217 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77a0adbb32b479199693550f4bd0e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.567344 13245 raft_consensus.cc:348] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.567572 13245 raft_consensus.cc:374] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.567624 13245 raft_consensus.cc:729] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.567488 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de82135b9e3745808395a881cf67e539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.567814 13245 consensus_queue.cc:260] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.568373 13245 ts_tablet_manager.cc:1398] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.208s user 0.000s sys 0.000s
I20240506 19:56:57.568681 13245 tablet_bootstrap.cc:492] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:57.572216 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebecd797899b41e2b4b00e0485922add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.572553 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858d58e1b2d64f47a097bb5eb7a236c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:57.581055 13242 ts_tablet_manager.cc:1888] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.581207 13242 log.cc:1196] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:57.593578 13242 ts_tablet_manager.cc:1909] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.594440 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 6b59ad067c274fddbba5f71e89480dec (table default.Table_37e9aef6398b47059569bab0a7265a94 [id=d8c76c1f826d49c4818b1e278d407b0e]) successfully deleted
W20240506 19:56:57.545065 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 128618us
I20240506 19:56:57.597463 13242 tablet_replica.cc:330] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.597848 13242 raft_consensus.cc:2229] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.598130 13242 raft_consensus.cc:2258] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.545254 14383 raft_consensus.cc:2781] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.598856 14682 raft_consensus.cc:482] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.598912 14383 raft_consensus.cc:686] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.598894 13242 ts_tablet_manager.cc:1875] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.599210 14383 consensus_queue.cc:237] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.599004 14682 raft_consensus.cc:504] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.599742 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 237143us
I20240506 19:56:57.600108 14682 leader_election.cc:288] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:57.601598 12783 leader_election.cc:341] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:57.603525 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.608816 14702 raft_consensus.cc:482] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.609014 14702 raft_consensus.cc:504] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.609675 14702 leader_election.cc:288] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.610304 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d53f3daaa2949a2b6883435f031dc37" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.610899 12921 leader_election.cc:341] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.611485 14702 raft_consensus.cc:482] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.611615 14702 raft_consensus.cc:504] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.612169 14702 leader_election.cc:288] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.612711 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.613219 12921 leader_election.cc:341] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.614188 12842 raft_consensus.cc:3037] T 3bb5a12521044102ae55ca953f1496f9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.616128 14712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:57.616376 14712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:57.617841 12531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:57.626135 13242 ts_tablet_manager.cc:1888] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.626288 13242 log.cc:1196] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/1615f6167aae4dc096b9f563ab5e6215
I20240506 19:56:57.626644 13242 ts_tablet_manager.cc:1909] T 1615f6167aae4dc096b9f563ab5e6215 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.627360 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 1615f6167aae4dc096b9f563ab5e6215 (table default.Table_feb73e349178483fb087f43a269605b0 [id=b19563cb49834a49924bb9d06b60b6ed]) successfully deleted
W20240506 19:56:57.631114 12922 leader_election.cc:341] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.631253 12922 leader_election.cc:302] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.631466 14702 raft_consensus.cc:2726] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.631788 14702 raft_consensus.cc:482] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.631907 14702 raft_consensus.cc:504] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.632499 14702 leader_election.cc:288] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.633100 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e04e93a5543f9934077006eaf2ba9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.633450 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4630abf54f024da5be5b45efccc0051d" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:56:57.633659 12921 leader_election.cc:341] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.634444 13244 tablet_bootstrap.cc:492] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.634590 13244 ts_tablet_manager.cc:1367] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.289s user 0.007s sys 0.006s
I20240506 19:56:57.635360 13244 raft_consensus.cc:348] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.635557 13244 raft_consensus.cc:374] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.635635 13244 raft_consensus.cc:729] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.635845 13244 consensus_queue.cc:260] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.636497 13244 ts_tablet_manager.cc:1398] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:57.636775 13244 tablet_bootstrap.cc:492] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.645897 13242 tablet_replica.cc:330] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.646123 13242 raft_consensus.cc:2229] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.646281 13242 raft_consensus.cc:2258] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.648504 13245 tablet_bootstrap.cc:654] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.650295 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46916e778bf34188959e1cfff084bffc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:57.650885 12921 leader_election.cc:341] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.651156 14459 leader_election.cc:288] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:57.651665 12922 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f3a8531a1793404699532295164a3895
I20240506 19:56:57.651960 12922 leader_election.cc:302] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.652429 14702 raft_consensus.cc:2726] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.653424 13242 ts_tablet_manager.cc:1875] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.654966 14721 tablet.cc:1722] T 4973e600038742a6bec0bb2d8f3551fe P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240506 19:56:57.657548 12781 leader_election.cc:341] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2c96f014293242be9640515697c0733b
I20240506 19:56:57.657666 12781 leader_election.cc:302] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.658128 14682 raft_consensus.cc:2726] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:57.659068 12781 leader_election.cc:341] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 6b59ad067c274fddbba5f71e89480dec
I20240506 19:56:57.659175 12781 leader_election.cc:302] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:56:57.659328 12781 raft_consensus.cc:2670] T 6b59ad067c274fddbba5f71e89480dec P ef18e88b87064ed4b6c20f9325ebcc33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240506 19:56:57.659551 13242 ts_tablet_manager.cc:1888] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.659646 13242 log.cc:1196] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/472c63d8995346d78633154a8f0d4c2b
I20240506 19:56:57.659955 13242 ts_tablet_manager.cc:1909] T 472c63d8995346d78633154a8f0d4c2b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:57.660394 12781 leader_election.cc:341] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:57.660507 12781 leader_election.cc:302] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.660526 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 472c63d8995346d78633154a8f0d4c2b (table default.Table_b122543bad9446f9bd6b57a8336b17bc [id=465e454733ed4ae4874968428ed4da60]) successfully deleted
I20240506 19:56:57.660898 14682 raft_consensus.cc:2726] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.667332 13244 tablet_bootstrap.cc:654] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:57.670397 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet acf5e347bb68466e811259132e3c99b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acf5e347bb68466e811259132e3c99b1 already in progress: creating tablet
W20240506 19:56:57.672159 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 93d6e4b9f0314cf19bbdd9fd9038d97d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d6e4b9f0314cf19bbdd9fd9038d97d already in progress: creating tablet
W20240506 19:56:57.672803 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 9be3b844a9334cfabea3946844ae1af8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9be3b844a9334cfabea3946844ae1af8 already in progress: creating tablet
W20240506 19:56:57.673408 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 392e4e4fa399443cb23b5641ff4c6c39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 392e4e4fa399443cb23b5641ff4c6c39 already in progress: creating tablet
W20240506 19:56:57.674057 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bac098377eea4798b7a5516a2a7bb5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bac098377eea4798b7a5516a2a7bb5b0 already in progress: creating tablet
I20240506 19:56:57.676729 13498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
W20240506 19:56:57.680442 12922 leader_election.cc:341] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1d53f3daaa2949a2b6883435f031dc37
I20240506 19:56:57.680584 12922 leader_election.cc:302] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.681195 14702 raft_consensus.cc:2726] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:57.681972 12922 leader_election.cc:341] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 373e04e93a5543f9934077006eaf2ba9
I20240506 19:56:57.682089 12922 leader_election.cc:302] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.673564 12781 leader_election.cc:302] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.674814 13242 tablet_replica.cc:330] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.682545 13242 raft_consensus.cc:2229] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.682628 14682 raft_consensus.cc:2726] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.683045 13242 raft_consensus.cc:2258] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.683418 14459 raft_consensus.cc:2726] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.684520 13242 ts_tablet_manager.cc:1875] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.689747 13244 tablet_bootstrap.cc:492] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.689977 13244 ts_tablet_manager.cc:1367] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.004s
I20240506 19:56:57.690990 13244 raft_consensus.cc:348] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.691253 13244 raft_consensus.cc:374] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.691383 13244 raft_consensus.cc:729] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.691704 13244 consensus_queue.cc:260] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.692324 13244 ts_tablet_manager.cc:1398] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:57.693260 13244 tablet_bootstrap.cc:492] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.695477 13244 tablet_bootstrap.cc:654] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.656303 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.674655 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46916e778bf34188959e1cfff084bffc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.679500 14722 tablet.cc:1722] T 4973e600038742a6bec0bb2d8f3551fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:57.701695 13116 raft_consensus.cc:2454] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.701481 13120 raft_consensus.cc:3037] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.702394 12922 leader_election.cc:302] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.702824 14459 raft_consensus.cc:2781] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.702891 14459 raft_consensus.cc:482] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.702944 14459 raft_consensus.cc:3037] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.705566 14459 raft_consensus.cc:504] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.705922 13120 raft_consensus.cc:2454] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.706746 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:57.707638 12508 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.253:36819, user_credentials={real_user=slave}} blocked reactor thread for 43609.2us
I20240506 19:56:57.710848 14459 leader_election.cc:288] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.711416 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 46916e778bf34188959e1cfff084bffc
I20240506 19:56:57.711961 12520 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33592 (request call id 1308) took 1233 ms. Trace:
I20240506 19:56:57.712056 12520 rpcz_store.cc:276] 0506 19:56:56.478297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.478333 (+ 36us) service_pool.cc:225] Handling call
0506 19:56:56.486764 (+ 8431us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x616000021d54. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 00007f8feed4915d 00007f8feed49253 00007f8feba7ad41 00007f8ff366734d 00007f8ff37f4ca6 00007f8fea7e6f66 00007f8fea7e6c95 00007f8fe6f1104e 00007f8fe6f0ff2e 00007f8fe6f16405 00007f8fe6f16d2f
0506 19:56:56.487053 (+ 289us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0506 19:56:56.487170 (+ 117us) catalog_manager.cc:2749] Modifying in-memory table state
0506 19:56:56.487224 (+ 54us) catalog_manager.cc:2758] Locking tablets
0506 19:56:57.343046 (+855822us) catalog_manager.cc:2772] Removing table and tablets from system table
0506 19:56:57.711219 (+368173us) catalog_manager.cc:2790] Removing table from by-name map
0506 19:56:57.711320 (+ 101us) catalog_manager.cc:2806] Committing in-memory state
0506 19:56:57.711381 (+ 61us) catalog_manager.cc:2810] Aborting table tasks
0506 19:56:57.711907 (+ 526us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:57.356973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:57.357044 (+ 71us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:57.357049 (+ 5us) write_op.cc:435] Acquired schema lock
0506 19:56:57.357050 (+ 1us) tablet.cc:660] Decoding operations
0506 19:56:57.357134 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:57.357141 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:57.357145 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0506 19:56:57.357222 (+ 77us) tablet.cc:699] Row locks acquired
0506 19:56:57.357224 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:57.357248 (+ 24us) write_op.cc:270] Start()
0506 19:56:57.357286 (+ 38us) write_op.cc:276] Timestamp: P: 1715025417357354 usec, L: 0
0506 19:56:57.364917 (+ 7631us) op_driver.cc:331] REPLICATION: starting
0506 19:56:57.365154 (+ 237us) log.cc:842] Serialized 762 byte log entry
0506 19:56:57.708151 (+342997us) op_driver.cc:444] REPLICATION: finished
0506 19:56:57.708664 (+ 513us) write_op.cc:301] APPLY: starting
0506 19:56:57.708687 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:57.708740 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:57.708765 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:57.708845 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:57.708882 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0506 19:56:57.708890 (+ 8us) write_op.cc:312] APPLY: finished
0506 19:56:57.708978 (+ 88us) log.cc:842] Serialized 29 byte log entry
0506 19:56:57.709045 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.709068 (+ 23us) write_op.cc:454] Released schema lock
0506 19:56:57.709075 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:57.709129 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.709133 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18928384,"child_traces":[["op",{"apply.queue_time_us":451,"apply.run_cpu_time_us":526,"apply.run_wall_time_us":527,"num_ops":2,"prepare.queue_time_us":13195,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":8240,"replication_time_us":343204}]]}
I20240506 19:56:57.714321 13242 ts_tablet_manager.cc:1888] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.714493 13242 log.cc:1196] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/2c96f014293242be9640515697c0733b
I20240506 19:56:57.714840 13242 ts_tablet_manager.cc:1909] T 2c96f014293242be9640515697c0733b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.715643 12842 raft_consensus.cc:3037] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.716838 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2a09a031cefc4757938ad6047310bc72" } new_table_name: "default.Table_a4962437b4904f948a5ceaf065c7ab3e" modify_external_catalogs: true
I20240506 19:56:57.718097 12513 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164] alter complete (version 1)
I20240506 19:56:57.719404 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6c429083621d42058b514a513fceff77" } modify_external_catalogs: true
I20240506 19:56:57.719588 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_6c429083621d42058b514a513fceff77" } modify_external_catalogs: true
I20240506 19:56:57.720664 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 2e840b9c72c142f5a8049f223e89ea5a
I20240506 19:56:57.722200 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 2c96f014293242be9640515697c0733b (table default.Table_b41fdc2ddabb43b6abd4d24fecd7447a [id=f67e8f35a154427f95ae0f771099e941]) successfully deleted
W20240506 19:56:57.722493 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet f476f745e0784f7f9886484367dd8c22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f476f745e0784f7f9886484367dd8c22 already in progress: creating tablet
I20240506 19:56:57.725641 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 46916e778bf34188959e1cfff084bffc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.725929 13246 tablet_replica.cc:330] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:57.726152 13246 raft_consensus.cc:2229] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.726316 13246 raft_consensus.cc:2258] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.726259 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfb78f8b470b46f485886ae5bd7833f6" } modify_external_catalogs: true
I20240506 19:56:57.726460 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_dfb78f8b470b46f485886ae5bd7833f6" } modify_external_catalogs: true
I20240506 19:56:57.726809 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.726889 13246 ts_tablet_manager.cc:1875] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.726990 13120 raft_consensus.cc:3037] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.730252 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_70f25b0c770549f5b47f5ac2d70cd33c" } modify_external_catalogs: true
W20240506 19:56:57.714849 12921 leader_election.cc:341] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.714970 14702 raft_consensus.cc:482] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.730449 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_70f25b0c770549f5b47f5ac2d70cd33c" } modify_external_catalogs: true
I20240506 19:56:57.730564 14702 raft_consensus.cc:504] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.731223 14702 leader_election.cc:288] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.719213 12922 leader_election.cc:302] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.727335 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 46916e778bf34188959e1cfff084bffc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.731781 14557 tablet_replica.cc:330] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.731972 14557 raft_consensus.cc:2229] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.732082 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e840b9c72c142f5a8049f223e89ea5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.732120 14557 raft_consensus.cc:2258] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.732196 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.732419 13116 raft_consensus.cc:2454] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.732689 14557 ts_tablet_manager.cc:1875] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.733253 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.733413 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e840b9c72c142f5a8049f223e89ea5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.733681 12809 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9296bb1104b4cbe279eecd3683796 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:47 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:57.733865 12921 leader_election.cc:341] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.734143 12922 leader_election.cc:302] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.734241 12814 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f4861a01404663870ce91978c8af4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.734539 14702 raft_consensus.cc:2781] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.734622 14702 raft_consensus.cc:482] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.734696 14702 raft_consensus.cc:3037] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.734982 13246 ts_tablet_manager.cc:1888] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.735081 13246 log.cc:1196] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/46916e778bf34188959e1cfff084bffc
I20240506 19:56:57.735467 13246 ts_tablet_manager.cc:1909] T 46916e778bf34188959e1cfff084bffc P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:57.736462 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df3ab9dff5148f9bb335a7209b29bbc with delete_type TABLET_DATA_DELETED (replaced by tablet efc41afd84a74652a82cc639bf562d99) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.736639 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 46916e778bf34188959e1cfff084bffc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.737107 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e90b93023e40b3bafbddf7bbcc10b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.737563 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 46916e778bf34188959e1cfff084bffc (table default.Table_fbd2d442ecdb4935bdff3a94b68cc073 [id=df889494c5f34752bf16bde1b5a80964]) successfully deleted
I20240506 19:56:57.737636 13242 tablet_replica.cc:330] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.737761 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e840b9c72c142f5a8049f223e89ea5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.737828 13242 raft_consensus.cc:2229] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.737979 13242 raft_consensus.cc:2258] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.738479 13242 ts_tablet_manager.cc:1875] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.737140 14702 raft_consensus.cc:504] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.742278 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 2e840b9c72c142f5a8049f223e89ea5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e840b9c72c142f5a8049f223e89ea5a already in progress: creating tablet
I20240506 19:56:57.742805 12848 raft_consensus.cc:3037] T 07c7c7b2da9e403f8241a512e210c4d9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.747958 14557 ts_tablet_manager.cc:1888] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.748095 14557 log.cc:1196] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/46916e778bf34188959e1cfff084bffc
I20240506 19:56:57.748495 14557 ts_tablet_manager.cc:1909] T 46916e778bf34188959e1cfff084bffc P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.749656 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 46916e778bf34188959e1cfff084bffc (table default.Table_fbd2d442ecdb4935bdff3a94b68cc073 [id=df889494c5f34752bf16bde1b5a80964]) successfully deleted
I20240506 19:56:57.749948 13242 ts_tablet_manager.cc:1888] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.750074 13242 log.cc:1196] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/8eb9296bb1104b4cbe279eecd3683796
I20240506 19:56:57.750365 13242 ts_tablet_manager.cc:1909] T 8eb9296bb1104b4cbe279eecd3683796 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.750447 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:57.750964 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 8eb9296bb1104b4cbe279eecd3683796 (table default.Table_970886a5e1a34021a2a6acf328821076 [id=cbfe8c554755404298262cddb5e7c011]) successfully deleted
W20240506 19:56:57.751070 12921 leader_election.cc:341] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.751220 14702 leader_election.cc:288] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.751394 13242 tablet_replica.cc:330] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.751632 13242 raft_consensus.cc:2229] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.751866 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.752075 13116 raft_consensus.cc:3037] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:57.752246 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 2e840b9c72c142f5a8049f223e89ea5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e840b9c72c142f5a8049f223e89ea5a already in progress: creating tablet
I20240506 19:56:57.752528 13242 raft_consensus.cc:2258] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.753046 13242 ts_tablet_manager.cc:1875] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.753654 14682 raft_consensus.cc:482] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.753775 14682 raft_consensus.cc:504] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.754294 13249 tablet_bootstrap.cc:492] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:57.754419 13249 ts_tablet_manager.cc:1367] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.219s user 0.040s sys 0.089s
I20240506 19:56:57.754446 14682 leader_election.cc:288] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:57.755187 13249 raft_consensus.cc:348] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.755316 13249 raft_consensus.cc:374] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.755352 13249 raft_consensus.cc:729] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:57.755609 12783 leader_election.cc:341] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ad0f74c7c4cd42d293fa244de3fd693b
I20240506 19:56:57.755532 13249 consensus_queue.cc:260] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.755863 13116 raft_consensus.cc:2454] T c659d848fa34442fb605f1757571b5e7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.755957 13249 ts_tablet_manager.cc:1398] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 19:56:57.756193 13249 tablet_bootstrap.cc:492] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:57.756305 12922 leader_election.cc:302] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.756850 14459 raft_consensus.cc:482] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:56:57.757138 12781 leader_election.cc:341] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ad0f74c7c4cd42d293fa244de3fd693b
I20240506 19:56:57.757244 12781 leader_election.cc:302] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:57.757187 14459 raft_consensus.cc:504] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.756845 14383 raft_consensus.cc:2781] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.757660 14383 raft_consensus.cc:686] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.757941 14459 leader_election.cc:288] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.757887 14383 consensus_queue.cc:237] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.758034 13249 tablet_bootstrap.cc:654] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.758549 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b803ae289344f88b604a2053881d59b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.758742 13116 raft_consensus.cc:2454] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.758850 14682 raft_consensus.cc:2726] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.759313 12922 leader_election.cc:302] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:57.759532 14707 raft_consensus.cc:2781] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.759605 14707 raft_consensus.cc:482] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.759667 14707 raft_consensus.cc:3037] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.759857 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b803ae289344f88b604a2053881d59b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.760664 12921 leader_election.cc:341] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.762080 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 2e840b9c72c142f5a8049f223e89ea5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e840b9c72c142f5a8049f223e89ea5a already in progress: creating tablet
I20240506 19:56:57.762117 14707 raft_consensus.cc:504] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.762688 14707 leader_election.cc:288] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.764029 13245 tablet_bootstrap.cc:492] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:57.764159 13245 ts_tablet_manager.cc:1367] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.196s user 0.003s sys 0.000s
I20240506 19:56:57.764201 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b803ae289344f88b604a2053881d59b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.764386 13124 raft_consensus.cc:3037] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.764976 13245 raft_consensus.cc:348] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.765156 13245 raft_consensus.cc:374] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.765205 13245 raft_consensus.cc:729] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.765477 13245 consensus_queue.cc:260] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.767141 13124 raft_consensus.cc:2454] T 7b803ae289344f88b604a2053881d59b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.767258 13242 ts_tablet_manager.cc:1888] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.767364 13242 log.cc:1196] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/99f4861a01404663870ce91978c8af4f
I20240506 19:56:57.768065 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b803ae289344f88b604a2053881d59b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:57.768738 12921 leader_election.cc:341] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.769125 12922 leader_election.cc:302] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.769706 14700 raft_consensus.cc:2781] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.769819 14700 raft_consensus.cc:686] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.770042 14700 consensus_queue.cc:237] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.770319 13242 ts_tablet_manager.cc:1909] T 99f4861a01404663870ce91978c8af4f P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.770954 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 99f4861a01404663870ce91978c8af4f (table default.Table_e091d2a1ea93426dacdda3d117da60b0 [id=869c8b683b3b447f87635fd3a00c3c4d]) successfully deleted
I20240506 19:56:57.771346 13242 tablet_replica.cc:330] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.771598 13242 raft_consensus.cc:2229] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.771760 13242 raft_consensus.cc:2258] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.772361 13245 ts_tablet_manager.cc:1398] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:57.772437 13242 ts_tablet_manager.cc:1875] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.772791 13245 tablet_bootstrap.cc:492] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:57.774482 13245 tablet_bootstrap.cc:654] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:57.676060 14712 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:56:57.677583 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
W20240506 19:56:57.776899 14081 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.253:36819, user_credentials={real_user=slave}} blocked reactor thread for 99402us
I20240506 19:56:57.779718 14712 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:57.782538 14617 tablet.cc:1722] T 4630abf54f024da5be5b45efccc0051d P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:57.774605 14639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 19:56:57.792513 14707 raft_consensus.cc:482] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.792683 14707 raft_consensus.cc:504] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.793483 13242 ts_tablet_manager.cc:1888] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.793617 13242 log.cc:1196] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/9df3ab9dff5148f9bb335a7209b29bbc
I20240506 19:56:57.793792 14707 leader_election.cc:288] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.793973 13242 ts_tablet_manager.cc:1909] T 9df3ab9dff5148f9bb335a7209b29bbc P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
W20240506 19:56:57.794638 12922 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d43da1d12ce647dcbcd31263932b2f54
I20240506 19:56:57.797107 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 9df3ab9dff5148f9bb335a7209b29bbc (table default.Table_24285783bcc5438e9af42796d83db179 [id=6b539c4e5b2444daa0f9ae05ed98ff54]) successfully deleted
W20240506 19:56:57.797945 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 46916e778bf34188959e1cfff084bffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46916e778bf34188959e1cfff084bffc already in progress: creating tablet
I20240506 19:56:57.798307 13242 tablet_replica.cc:330] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.798524 13242 raft_consensus.cc:2229] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.798703 13242 raft_consensus.cc:2258] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.799337 13242 ts_tablet_manager.cc:1875] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.806537 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43da1d12ce647dcbcd31263932b2f54" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.807255 12921 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.807394 12921 leader_election.cc:302] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.809355 14460 raft_consensus.cc:2726] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.812585 13120 raft_consensus.cc:2454] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.813103 12922 leader_election.cc:302] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.813395 14707 raft_consensus.cc:2781] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.813515 14707 raft_consensus.cc:686] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.813915 14707 consensus_queue.cc:237] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.821261 12518 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41853001e53948a49fbb75b093dda93d consensus state has no leader
I20240506 19:56:57.823479 12518 catalog_manager.cc:5576] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475) does not have the latest schema for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]). Expected version 2 got 0
W20240506 19:56:57.824165 12517 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd1f00e8620640b08cc054c043900738 consensus state has no leader
I20240506 19:56:57.824596 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_f05ee8e1debb411c9f6f7be6e92288fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.825150 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f05ee8e1debb411c9f6f7be6e92288fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.826824 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_9a8bd42081bd47fdb4e31ba99d96756e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.827344 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a8bd42081bd47fdb4e31ba99d96756e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.827680 13242 ts_tablet_manager.cc:1888] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:57.827808 13242 log.cc:1196] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/a9e90b93023e40b3bafbddf7bbcc10b2
I20240506 19:56:57.828106 13242 ts_tablet_manager.cc:1909] T a9e90b93023e40b3bafbddf7bbcc10b2 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.828804 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet a9e90b93023e40b3bafbddf7bbcc10b2 (table default.Table_6b562516d9b845789bf9d901c015e37c [id=11d0ad6e93ef421182aaac061edb8501]) successfully deleted
I20240506 19:56:57.833993 14697 raft_consensus.cc:482] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.834141 14697 raft_consensus.cc:504] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.834733 14697 leader_election.cc:288] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:57.835354 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b04f4a9ca51d452789f5dc4bd36f5c6f consensus state has no leader
I20240506 19:56:57.835752 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_1403b3f7ba884b308e7346ed7eaa06c6" } new_table_name: "default.Table_c7f65b0de80a4c4cbd8a85f397e7a344" modify_external_catalogs: true
I20240506 19:56:57.837023 12519 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d2bd6c448e9448b38dd12bc3148207b8
I20240506 19:56:57.837342 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ecc75bab446e39c687f3c8b740361" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.837419 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_3380422dfa66489eb9f22c0b76154c8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.837878 12921 leader_election.cc:341] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.837929 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3380422dfa66489eb9f22c0b76154c8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.838131 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ecc75bab446e39c687f3c8b740361" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.838313 13120 raft_consensus.cc:2454] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.838634 12922 leader_election.cc:302] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.839287 14697 raft_consensus.cc:2781] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.839372 14697 raft_consensus.cc:482] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.839387 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet a564c22b68f74b92ba0f313fcb29c2f7
I20240506 19:56:57.839432 14697 raft_consensus.cc:3037] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.840154 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2cc2d44f3f26460e8b237e82572aab2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.840648 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cc2d44f3f26460e8b237e82572aab2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.841754 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 9f47449a5a504f939e131743d94433ee
I20240506 19:56:57.842082 14697 raft_consensus.cc:504] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.842332 12514 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b89fb24d94042e1bfa772265a991680 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:57.842963 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_5eb85e351b7645f5925a458693f85835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:57.843189 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ecc75bab446e39c687f3c8b740361" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:57.843487 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5eb85e351b7645f5925a458693f85835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.843554 13116 raft_consensus.cc:3037] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.845423 14697 leader_election.cc:288] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.845472 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f47449a5a504f939e131743d94433ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.845472 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet a564c22b68f74b92ba0f313fcb29c2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:57.845759 13242 tablet_replica.cc:330] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:57.845966 13242 raft_consensus.cc:2229] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.846199 13242 raft_consensus.cc:2258] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.846170 14459 raft_consensus.cc:482] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.846256 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f47449a5a504f939e131743d94433ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.846349 14459 raft_consensus.cc:504] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.846450 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet a564c22b68f74b92ba0f313fcb29c2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:57.846491 13246 tablet_replica.cc:330] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:57.846659 13246 raft_consensus.cc:2229] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:57.846827 13242 ts_tablet_manager.cc:1875] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.846837 13246 raft_consensus.cc:2258] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.846943 14459 leader_election.cc:288] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.847052 14459 raft_consensus.cc:482] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.847138 14459 raft_consensus.cc:504] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.847381 13246 ts_tablet_manager.cc:1875] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.847631 14459 leader_election.cc:288] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.849171 13116 raft_consensus.cc:2454] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.850317 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f47449a5a504f939e131743d94433ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.850387 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff7050bbc2e47cf823d25ec10b89718" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.850570 14557 tablet_replica.cc:330] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:57.850569 13124 raft_consensus.cc:2454] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.850752 14557 raft_consensus.cc:2229] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:57.850827 14460 raft_consensus.cc:482] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.850925 14460 raft_consensus.cc:504] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.851023 14557 raft_consensus.cc:2258] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:57.851420 14460 leader_election.cc:288] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.851562 14557 ts_tablet_manager.cc:1875] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:57.851742 12922 leader_election.cc:302] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:57.852141 14460 raft_consensus.cc:2781] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.852244 14460 raft_consensus.cc:686] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.852465 14460 consensus_queue.cc:237] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.853858 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f382a89f94e4fdf9b8266d049635329" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.854030 13124 raft_consensus.cc:2454] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:57.854269 12922 leader_election.cc:302] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:57.854580 14459 raft_consensus.cc:2781] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.854643 14459 raft_consensus.cc:482] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.854697 14459 raft_consensus.cc:3037] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.850541 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet a564c22b68f74b92ba0f313fcb29c2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:57.857949 12922 leader_election.cc:302] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:57.858376 14697 raft_consensus.cc:2781] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.858582 14697 raft_consensus.cc:482] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.858587 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b022c086d141a19a9b35c4a8c8bef9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.858851 14697 raft_consensus.cc:3037] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.859309 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f382a89f94e4fdf9b8266d049635329" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:57.859776 12921 leader_election.cc:341] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.857911 14557 ts_tablet_manager.cc:1888] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:57.861460 14557 log.cc:1196] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/9f47449a5a504f939e131743d94433ee
I20240506 19:56:57.863299 14557 ts_tablet_manager.cc:1909] T 9f47449a5a504f939e131743d94433ee P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:57.864946 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]) successfully deleted
W20240506 19:56:57.866030 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet a564c22b68f74b92ba0f313fcb29c2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a564c22b68f74b92ba0f313fcb29c2f7 already in progress: creating tablet
I20240506 19:56:57.866078 14702 consensus_queue.cc:1035] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:57.866919 14697 raft_consensus.cc:504] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.869490 14697 leader_election.cc:288] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.869880 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f382a89f94e4fdf9b8266d049635329" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.870076 13111 raft_consensus.cc:3037] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.873868 13246 ts_tablet_manager.cc:1888] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:57.873996 13246 log.cc:1196] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/9f47449a5a504f939e131743d94433ee
I20240506 19:56:57.874230 13111 raft_consensus.cc:2454] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.874387 13246 ts_tablet_manager.cc:1909] T 9f47449a5a504f939e131743d94433ee P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:57.874763 12922 leader_election.cc:302] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:57.877689 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]) successfully deleted
I20240506 19:56:57.877915 14702 raft_consensus.cc:2781] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
W20240506 19:56:57.878048 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet a564c22b68f74b92ba0f313fcb29c2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a564c22b68f74b92ba0f313fcb29c2f7 already in progress: creating tablet
I20240506 19:56:57.879689 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f382a89f94e4fdf9b8266d049635329" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:57.880548 12921 leader_election.cc:341] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.880892 12921 leader_election.cc:341] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.881716 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b022c086d141a19a9b35c4a8c8bef9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.857144 14459 raft_consensus.cc:504] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.883718 14459 leader_election.cc:288] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:57.884792 12922 leader_election.cc:341] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.884907 12922 leader_election.cc:302] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.885277 14738 raft_consensus.cc:2726] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.886214 14702 raft_consensus.cc:686] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.886519 14702 consensus_queue.cc:237] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.887604 14707 raft_consensus.cc:482] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.887771 14707 raft_consensus.cc:504] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.888849 14707 leader_election.cc:288] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.889011 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff7050bbc2e47cf823d25ec10b89718" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.889204 13116 raft_consensus.cc:3037] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.889950 13242 ts_tablet_manager.cc:1888] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:57.890074 13242 log.cc:1196] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/9f47449a5a504f939e131743d94433ee
I20240506 19:56:57.890409 13242 ts_tablet_manager.cc:1909] T 9f47449a5a504f939e131743d94433ee P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:57.891047 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet 9f47449a5a504f939e131743d94433ee (table default.Table_dfb78f8b470b46f485886ae5bd7833f6 [id=6fcb7b163e714ce7bf9e991838356b50]) successfully deleted
W20240506 19:56:57.891870 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet a564c22b68f74b92ba0f313fcb29c2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a564c22b68f74b92ba0f313fcb29c2f7 already in progress: creating tablet
I20240506 19:56:57.892154 13116 raft_consensus.cc:2454] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:57.895220 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff7050bbc2e47cf823d25ec10b89718" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:57.895887 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff7050bbc2e47cf823d25ec10b89718" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.898223 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.898767 14739 raft_consensus.cc:482] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:57.898911 14739 raft_consensus.cc:504] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.899435 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:57.899482 14740 consensus_queue.cc:1035] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:57.899693 13114 raft_consensus.cc:2454] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:57.898783 12921 consensus_peers.cc:485] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:57.900710 12921 leader_election.cc:341] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.901082 12921 leader_election.cc:341] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.901559 12922 leader_election.cc:302] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:57.901736 12921 leader_election.cc:341] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.901825 14740 raft_consensus.cc:2781] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:57.902313 14740 raft_consensus.cc:482] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:57.902470 14740 raft_consensus.cc:3037] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.907366 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ecc75bab446e39c687f3c8b740361" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:57.908946 12922 leader_election.cc:341] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bece304e0ba048ecbb2d92aa71fdf707
I20240506 19:56:57.909544 14739 leader_election.cc:288] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.909651 14740 raft_consensus.cc:504] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.909996 12921 leader_election.cc:341] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.910442 14740 leader_election.cc:288] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:57.911315 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bece304e0ba048ecbb2d92aa71fdf707" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:57.911384 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:57.911617 13114 raft_consensus.cc:3037] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:57.914912 13114 raft_consensus.cc:2454] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:57.916030 12921 leader_election.cc:341] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:57.916203 12921 leader_election.cc:302] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.916882 14702 raft_consensus.cc:2726] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:57.917104 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:57.917322 14721 tablet.cc:1722] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240506 19:56:57.922734 12921 leader_election.cc:341] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:57.923980 12921 consensus_peers.cc:485] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:57.925099 14617 tablet.cc:1722] T 45b2a4720f6a46e4b0412a58c1e26bf7 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240506 19:56:57.926853 14702 consensus_queue.cc:1035] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:57.927891 12922 leader_election.cc:302] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:57.928285 14702 raft_consensus.cc:2781] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.928404 14702 raft_consensus.cc:686] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.928619 14702 consensus_queue.cc:237] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:57.938467 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 81293us
I20240506 19:56:57.945283 13244 tablet_bootstrap.cc:492] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:57.945457 13244 ts_tablet_manager.cc:1367] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.252s user 0.005s sys 0.002s
I20240506 19:56:57.946167 13244 raft_consensus.cc:348] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.946375 13244 raft_consensus.cc:374] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:57.946427 13244 raft_consensus.cc:729] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:57.946659 13244 consensus_queue.cc:260] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.949954 13244 ts_tablet_manager.cc:1398] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:57.950258 13244 tablet_bootstrap.cc:492] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:57.964264 12517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_92a91e3ad2a7411b8cdf06c227b34759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.964866 12517 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92a91e3ad2a7411b8cdf06c227b34759 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.966516 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_d7890a93b92a4c6d881598c0bb5f5a21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:57.966612 12922 leader_election.cc:302] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:56:57.967042 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7890a93b92a4c6d881598c0bb5f5a21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.967135 14700 raft_consensus.cc:2781] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:57.967445 14700 raft_consensus.cc:686] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:57.967698 14700 consensus_queue.cc:237] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:57.970710 14722 tablet.cc:1722] T 9d0d827d732e4261a7a8edb549c1c8e3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:57.970865 14721 tablet.cc:1722] T 9d0d827d732e4261a7a8edb549c1c8e3 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:57.980432 14722 tablet.cc:1722] T 45b2a4720f6a46e4b0412a58c1e26bf7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240506 19:56:57.982560 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ede01d0bed034c16bb580124bd64d241 consensus state has no leader
I20240506 19:56:57.983100 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_42386049e25d4c309be88fb22d88bc71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.983784 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42386049e25d4c309be88fb22d88bc71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.985579 12519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_badaf4211410493f99e8fd4813c8b489"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.986083 12519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_badaf4211410493f99e8fd4813c8b489 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:57.987404 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62d73cc2b6a847ebb8f40ad1ce877813" } modify_external_catalogs: true
I20240506 19:56:57.987623 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_62d73cc2b6a847ebb8f40ad1ce877813" } modify_external_catalogs: true
I20240506 19:56:57.989096 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_34e944977e794f829c328e4fc138baec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:57.989627 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34e944977e794f829c328e4fc138baec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.006268 13249 tablet_bootstrap.cc:492] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:58.009102 13249 ts_tablet_manager.cc:1367] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.253s user 0.006s sys 0.002s
I20240506 19:56:58.009948 13249 raft_consensus.cc:348] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.010188 13249 raft_consensus.cc:374] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.010380 13249 raft_consensus.cc:729] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.010629 13249 consensus_queue.cc:260] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.011154 13249 ts_tablet_manager.cc:1398] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:58.011444 13249 tablet_bootstrap.cc:492] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:58.013015 13249 tablet_bootstrap.cc:654] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.021347 13244 tablet_bootstrap.cc:654] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.022114 13245 tablet_bootstrap.cc:492] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.022356 13245 ts_tablet_manager.cc:1367] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.250s user 0.022s sys 0.029s
I20240506 19:56:58.023293 13245 raft_consensus.cc:348] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.023635 13245 raft_consensus.cc:374] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.023814 13245 raft_consensus.cc:729] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.024179 13245 consensus_queue.cc:260] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:58.031081 12921 consensus_peers.cc:485] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.038408 13245 ts_tablet_manager.cc:1398] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240506 19:56:58.038834 13245 tablet_bootstrap.cc:492] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:58.041465 13245 tablet_bootstrap.cc:654] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.041602 13244 tablet_bootstrap.cc:492] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.041743 13244 ts_tablet_manager.cc:1367] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.001s
I20240506 19:56:58.042609 13244 raft_consensus.cc:348] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.042881 13244 raft_consensus.cc:374] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.042949 13244 raft_consensus.cc:729] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.043195 13244 consensus_queue.cc:260] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.043833 13244 ts_tablet_manager.cc:1398] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:58.045061 13244 tablet_bootstrap.cc:492] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.048668 13244 tablet_bootstrap.cc:654] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.062322 13244 tablet_bootstrap.cc:492] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.062510 13244 ts_tablet_manager.cc:1367] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240506 19:56:58.063455 13244 raft_consensus.cc:348] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.063721 13244 raft_consensus.cc:374] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.063788 13244 raft_consensus.cc:729] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.064059 13244 consensus_queue.cc:260] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.064729 13244 ts_tablet_manager.cc:1398] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:58.068120 13244 tablet_bootstrap.cc:492] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.068295 14682 raft_consensus.cc:482] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.068428 14682 raft_consensus.cc:504] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.069048 14682 leader_election.cc:288] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.069813 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:58.070219 13244 tablet_bootstrap.cc:654] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.070132 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e748d2c32a4b09abc872fed318e643" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:58.070695 12781 leader_election.cc:302] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:58.071141 14682 raft_consensus.cc:2726] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:58.082716 13244 tablet_bootstrap.cc:492] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.082923 13244 ts_tablet_manager.cc:1367] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240506 19:56:58.083936 13244 raft_consensus.cc:348] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.084232 13244 raft_consensus.cc:374] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.084302 13244 raft_consensus.cc:729] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.087831 13249 tablet_bootstrap.cc:492] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:58.087991 13249 ts_tablet_manager.cc:1367] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.077s user 0.000s sys 0.002s
I20240506 19:56:58.088773 13249 raft_consensus.cc:348] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.088994 13249 raft_consensus.cc:374] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.089063 13249 raft_consensus.cc:729] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.089339 13249 consensus_queue.cc:260] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.090783 13249 ts_tablet_manager.cc:1398] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240506 19:56:58.091974 13249 tablet_bootstrap.cc:492] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:58.093856 13249 tablet_bootstrap.cc:654] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.109226 14700 consensus_queue.cc:1035] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:58.109670 13244 consensus_queue.cc:260] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.112715 12521 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_322cb621e5734d8bb9694e30ac509ea7" } modify_external_catalogs: true
I20240506 19:56:58.112989 12521 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_322cb621e5734d8bb9694e30ac509ea7" } modify_external_catalogs: true
I20240506 19:56:58.117285 14656 raft_consensus.cc:482] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.117465 14656 raft_consensus.cc:504] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.118191 14656 leader_election.cc:288] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.119812 13244 ts_tablet_manager.cc:1398] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.037s user 0.002s sys 0.002s
I20240506 19:56:58.120219 13244 tablet_bootstrap.cc:492] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.122056 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:58.122326 13245 tablet_bootstrap.cc:492] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.122581 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:58.122802 12987 raft_consensus.cc:2454] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:58.123023 13245 ts_tablet_manager.cc:1367] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.084s user 0.017s sys 0.043s
W20240506 19:56:58.124042 13058 leader_election.cc:341] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.124475 13057 leader_election.cc:302] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.124816 13245 raft_consensus.cc:348] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.128005 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 4dcc036a20824736b16815bcb47645c4 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:58.128662 12516 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_9f4ce0d827e5469ba860bae2d23245f5" } new_table_name: "default.Table_0deb4b5d182d46de8bece88de704f3ea" modify_external_catalogs: true
I20240506 19:56:58.131603 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_bd43d9d17e59466187a8090e48b210ef" } new_table_name: "default.Table_02b297f901384fd19bf275bcda8a2cab" modify_external_catalogs: true
I20240506 19:56:58.133149 12519 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4b829083fa8c4c29b7b0c730361bb0d9" } new_table_name: "default.Table_ddc8aae93ec243fd9f2aa7841c96bbca" modify_external_catalogs: true
I20240506 19:56:58.134230 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet ff722c670f654025b25ad483d22d8d25
I20240506 19:56:58.134969 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4ac15d4e3f6b4719b8b728be5474dd63" } new_table_name: "default.Table_fa78b5478d7c4ca08cb6dcbb0b0b349e" modify_external_catalogs: true
I20240506 19:56:58.136801 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet ff722c670f654025b25ad483d22d8d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.137610 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet ff722c670f654025b25ad483d22d8d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:41490
W20240506 19:56:58.138326 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet ff722c670f654025b25ad483d22d8d25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff722c670f654025b25ad483d22d8d25 already in progress: creating tablet
I20240506 19:56:58.139008 13249 tablet_bootstrap.cc:492] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:58.139186 13249 ts_tablet_manager.cc:1367] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240506 19:56:58.140350 13249 raft_consensus.cc:348] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.140550 13249 raft_consensus.cc:374] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.140604 13249 raft_consensus.cc:729] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:56:58.138891 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet ff722c670f654025b25ad483d22d8d25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff722c670f654025b25ad483d22d8d25 already in progress: creating tablet
I20240506 19:56:58.140856 13249 consensus_queue.cc:260] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.143888 13249 ts_tablet_manager.cc:1398] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240506 19:56:58.144234 13249 tablet_bootstrap.cc:492] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:58.149053 12921 consensus_peers.cc:485] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.124981 14656 raft_consensus.cc:2781] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:58.152171 14656 raft_consensus.cc:482] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:58.152269 14656 raft_consensus.cc:3037] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.155011 14656 raft_consensus.cc:504] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.136416 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet ff722c670f654025b25ad483d22d8d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:57 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.156389 14656 leader_election.cc:288] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:58.156495 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet ff722c670f654025b25ad483d22d8d25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff722c670f654025b25ad483d22d8d25 already in progress: creating tablet
I20240506 19:56:58.156879 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:58.157277 13245 raft_consensus.cc:374] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.157370 13245 raft_consensus.cc:729] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:58.157469 13058 leader_election.cc:341] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.157619 13245 consensus_queue.cc:260] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.158004 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:58.158229 12987 raft_consensus.cc:3037] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.158309 13245 ts_tablet_manager.cc:1398] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20240506 19:56:58.158560 13245 tablet_bootstrap.cc:492] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:58.160034 13245 tablet_bootstrap.cc:654] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.165531 12987 raft_consensus.cc:2454] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:58.167984 13057 leader_election.cc:302] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.168702 13244 tablet_bootstrap.cc:654] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:58.174422 12519 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45421a6499f2468dadf9fb10866b0144 has no consensus state
I20240506 19:56:58.174860 12519 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e050c5de47634fe69b12743c8b30d5e9" } modify_external_catalogs: true
I20240506 19:56:58.175108 12519 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e050c5de47634fe69b12743c8b30d5e9" } modify_external_catalogs: true
I20240506 19:56:58.176007 14757 raft_consensus.cc:2781] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:58.176290 14757 raft_consensus.cc:686] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:58.176589 14757 consensus_queue.cc:237] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:58.177614 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ebda423ac5a54fd6894369f417018b4d has no consensus state
I20240506 19:56:58.177672 13249 tablet_bootstrap.cc:654] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:58.178787 12515 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6eac98ac8f040c58f61106bb0e989de consensus state has no leader
I20240506 19:56:58.179348 12515 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2c27273f191c449d882ac1e69af5a4e0" } modify_external_catalogs: true
I20240506 19:56:58.179589 12515 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_2c27273f191c449d882ac1e69af5a4e0" } modify_external_catalogs: true
W20240506 19:56:58.181492 12516 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc4da6cfffd74edb8d34019662e430d6 consensus state has no leader
I20240506 19:56:58.181833 12516 master_service.cc:946] ReplaceTablet: received request to replace tablet 7390b6d4e5054374bed2ca2b4e4bd74f from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:58.182400 12514 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4dcb29fa5425483296ab644d95581803" } modify_external_catalogs: true
I20240506 19:56:58.182647 12514 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_4dcb29fa5425483296ab644d95581803" } modify_external_catalogs: true
I20240506 19:56:58.196727 13249 tablet_bootstrap.cc:492] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:58.196934 13249 ts_tablet_manager.cc:1367] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.003s
I20240506 19:56:58.198308 13249 raft_consensus.cc:348] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.198549 13249 raft_consensus.cc:374] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.198668 13249 raft_consensus.cc:729] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.201478 13249 consensus_queue.cc:260] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.202639 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 7b803ae289344f88b604a2053881d59b (table default.Table_6013611882be454da8ec1ea823261f6a [id=d2ffd5634f2042b7a85cbe74d6ca1df5]). Expected version 3 got 0
I20240506 19:56:58.202850 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (table default.Table_23747e8c1b314e4eaa4566b557389261 [id=a8b5f55150a54dcf8a19dc9e1153ea87]). Expected version 1 got 0
I20240506 19:56:58.203050 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 45da091c44824bd6bb8d390a5fb01ef8 (table default.Table_8afdb494805e4ba7838632b12912ec36 [id=22fff2da3af34564b794765f53773f81]). Expected version 2 got 0
I20240506 19:56:58.203439 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6ace0ec43ba44060bcec2a50e1f48098 (table default.Table_61becbd043ee470ca2ee3f4969d59971 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]). Expected version 2 got 0
I20240506 19:56:58.203773 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]). Expected version 1 got 0
I20240506 19:56:58.203977 12520 catalog_manager.cc:5576] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:58.204988 12818 tablet_service.cc:1467] Processing CreateTablet for tablet beccc4d2998f4229a9fa3507a7a090fc (DEFAULT_TABLE table=default.Table_629374150cbb4998973db994df329a42 [id=b12b185a8f664dada902a4448a7a29aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.205374 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 86b31abd5d31421898bc1d10739404d9 (DEFAULT_TABLE table=default.Table_2da22bc9de82467f80578b673f6faab6 [id=08b553c14b6743b38fda0abacb84766d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.206482 12950 tablet_service.cc:1467] Processing CreateTablet for tablet beccc4d2998f4229a9fa3507a7a090fc (DEFAULT_TABLE table=default.Table_629374150cbb4998973db994df329a42 [id=b12b185a8f664dada902a4448a7a29aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.206936 12950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beccc4d2998f4229a9fa3507a7a090fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.205459 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beccc4d2998f4229a9fa3507a7a090fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.209614 12957 tablet_service.cc:1467] Processing CreateTablet for tablet 86b31abd5d31421898bc1d10739404d9 (DEFAULT_TABLE table=default.Table_2da22bc9de82467f80578b673f6faab6 [id=08b553c14b6743b38fda0abacb84766d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.209998 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b31abd5d31421898bc1d10739404d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:58.213307 12508 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 45421a6499f2468dadf9fb10866b0144 (table default.Table_ddc8aae93ec243fd9f2aa7841c96bbca [id=505e004369be45e3950f15e43d47a14e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 45421a6499f2468dadf9fb10866b0144 consensus state has no leader
I20240506 19:56:58.213511 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b31abd5d31421898bc1d10739404d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.214161 13091 tablet_service.cc:1467] Processing CreateTablet for tablet 829421d3782e4ea6a748a28609b78ffd (DEFAULT_TABLE table=default.Table_9e229895c2804c09bd30a74901fd77b6 [id=38db8971145547068978042310ba1d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.214496 13092 tablet_service.cc:1467] Processing CreateTablet for tablet 32fec440f0cc4490a71eb37d6a1155ae (DEFAULT_TABLE table=default.Table_11ace0d42c9840d19108e3bfa5f1bb71 [id=29d018e8e77c4cbfa4cec78985b4de98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.214655 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829421d3782e4ea6a748a28609b78ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.214844 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fec440f0cc4490a71eb37d6a1155ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.215395 13090 tablet_service.cc:1467] Processing CreateTablet for tablet beccc4d2998f4229a9fa3507a7a090fc (DEFAULT_TABLE table=default.Table_629374150cbb4998973db994df329a42 [id=b12b185a8f664dada902a4448a7a29aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.215740 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beccc4d2998f4229a9fa3507a7a090fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.217525 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 86b31abd5d31421898bc1d10739404d9 (DEFAULT_TABLE table=default.Table_2da22bc9de82467f80578b673f6faab6 [id=08b553c14b6743b38fda0abacb84766d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.217814 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b31abd5d31421898bc1d10739404d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.217927 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce0a5be54ee4bafae89d59d29b0c8df (DEFAULT_TABLE table=default.Table_3eff05dacdb8400894bde49d2337d7f3 [id=a332c12a7dca471eb9e79b7974ea26a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.218209 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce0a5be54ee4bafae89d59d29b0c8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.221556 13103 tablet_service.cc:1467] Processing CreateTablet for tablet cf5029e26a394caab1ea238c3924b8d9 (DEFAULT_TABLE table=default.Table_02edd5e110a34ea292b65d41b0963109 [id=a18b40d9fb2a4e348f06bdb0ef822aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.221890 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5029e26a394caab1ea238c3924b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.223672 13098 tablet_service.cc:1467] Processing CreateTablet for tablet fee2e352f0da4e9e80e858a123c984da (DEFAULT_TABLE table=default.Table_096d0d9806624f8dab6c61cfd67e5481 [id=8f3a327c7cff4778ace0d6588fbba7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.223948 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee2e352f0da4e9e80e858a123c984da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.225519 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 3411f2ae4a4e4e1b8609292f7ac86c70 (DEFAULT_TABLE table=default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.225831 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3411f2ae4a4e4e1b8609292f7ac86c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.229139 13249 ts_tablet_manager.cc:1398] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240506 19:56:58.229281 13087 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0cdb80cd54bcf99a5eaab85a13363 (DEFAULT_TABLE table=default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.229779 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0cdb80cd54bcf99a5eaab85a13363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.233573 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3daa18d47842b089d2e94d29a3b21e (DEFAULT_TABLE table=default.Table_1590f38624f04f70a068b387db86e4bc [id=b21b01c7b4d146c5874a19d6621fb369]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.233897 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f3daa18d47842b089d2e94d29a3b21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.236675 13244 tablet_bootstrap.cc:492] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.237560 13096 tablet_service.cc:1467] Processing CreateTablet for tablet f501a21a1aa349118d47bdba73463816 (DEFAULT_TABLE table=default.Table_f162120d90f640e09da676cac7d0c187 [id=0b05c2baa470418c9f1c32a7de9167da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.237900 13096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f501a21a1aa349118d47bdba73463816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.241046 13249 tablet_bootstrap.cc:492] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:58.241638 13097 tablet_service.cc:1467] Processing CreateTablet for tablet 5b472ba7729c4f6184e07eff618407f4 (DEFAULT_TABLE table=default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.241994 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b472ba7729c4f6184e07eff618407f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.244992 13245 tablet_bootstrap.cc:492] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.245128 13245 ts_tablet_manager.cc:1367] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.013s
I20240506 19:56:58.245463 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 5cee27bc6e334dc3ae25f7ce5c40da00 (DEFAULT_TABLE table=default.Table_a0d74db25f05423f8b5ae3b82c7a54b1 [id=21294b97805743eb8444823b07a0ed7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.245827 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee27bc6e334dc3ae25f7ce5c40da00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.245970 13245 raft_consensus.cc:348] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.246138 13245 raft_consensus.cc:374] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.246199 13245 raft_consensus.cc:729] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.246567 13245 consensus_queue.cc:260] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.247854 13245 ts_tablet_manager.cc:1398] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 19:56:58.248097 13245 tablet_bootstrap.cc:492] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:58.249377 13244 ts_tablet_manager.cc:1367] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.000s
I20240506 19:56:58.250228 13244 raft_consensus.cc:348] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.250428 13244 raft_consensus.cc:374] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.250475 13244 raft_consensus.cc:729] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.251457 13249 tablet_bootstrap.cc:654] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.252568 13244 consensus_queue.cc:260] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.206485 12961 tablet_service.cc:1467] Processing CreateTablet for tablet 829421d3782e4ea6a748a28609b78ffd (DEFAULT_TABLE table=default.Table_9e229895c2804c09bd30a74901fd77b6 [id=38db8971145547068978042310ba1d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.253186 13244 ts_tablet_manager.cc:1398] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240506 19:56:58.253338 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829421d3782e4ea6a748a28609b78ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.254245 13244 tablet_bootstrap.cc:492] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.255766 13244 tablet_bootstrap.cc:654] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.258406 13245 tablet_bootstrap.cc:654] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.258455 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 3411f2ae4a4e4e1b8609292f7ac86c70 (DEFAULT_TABLE table=default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.258884 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3411f2ae4a4e4e1b8609292f7ac86c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.259686 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 7300ca61d4474d2183f4281909400897 (DEFAULT_TABLE table=default.Table_111e3f9425804b4ca0bf599cc9377867 [id=8026f48a5250456a8cea07bb6446b845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.259718 12816 tablet_service.cc:1467] Processing CreateTablet for tablet 32fec440f0cc4490a71eb37d6a1155ae (DEFAULT_TABLE table=default.Table_11ace0d42c9840d19108e3bfa5f1bb71 [id=29d018e8e77c4cbfa4cec78985b4de98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.259994 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7300ca61d4474d2183f4281909400897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.260282 12814 tablet_service.cc:1467] Processing CreateTablet for tablet cf5029e26a394caab1ea238c3924b8d9 (DEFAULT_TABLE table=default.Table_02edd5e110a34ea292b65d41b0963109 [id=a18b40d9fb2a4e348f06bdb0ef822aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.260592 12814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5029e26a394caab1ea238c3924b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.261509 12815 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0cdb80cd54bcf99a5eaab85a13363 (DEFAULT_TABLE table=default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.261688 12817 tablet_service.cc:1467] Processing CreateTablet for tablet f501a21a1aa349118d47bdba73463816 (DEFAULT_TABLE table=default.Table_f162120d90f640e09da676cac7d0c187 [id=0b05c2baa470418c9f1c32a7de9167da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.261813 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0cdb80cd54bcf99a5eaab85a13363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.261971 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f501a21a1aa349118d47bdba73463816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.262216 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3daa18d47842b089d2e94d29a3b21e (DEFAULT_TABLE table=default.Table_1590f38624f04f70a068b387db86e4bc [id=b21b01c7b4d146c5874a19d6621fb369]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.262512 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f3daa18d47842b089d2e94d29a3b21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.265573 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 5b472ba7729c4f6184e07eff618407f4 (DEFAULT_TABLE table=default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.265895 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b472ba7729c4f6184e07eff618407f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.268697 12842 raft_consensus.cc:3037] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.261519 12809 tablet_service.cc:1467] Processing CreateTablet for tablet fee2e352f0da4e9e80e858a123c984da (DEFAULT_TABLE table=default.Table_096d0d9806624f8dab6c61cfd67e5481 [id=8f3a327c7cff4778ace0d6588fbba7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.269553 12812 tablet_service.cc:1467] Processing CreateTablet for tablet 5cee27bc6e334dc3ae25f7ce5c40da00 (DEFAULT_TABLE table=default.Table_a0d74db25f05423f8b5ae3b82c7a54b1 [id=21294b97805743eb8444823b07a0ed7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.269707 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee2e352f0da4e9e80e858a123c984da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.269876 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee27bc6e334dc3ae25f7ce5c40da00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.273559 12824 tablet_service.cc:1467] Processing CreateTablet for tablet 45421a6499f2468dadf9fb10866b0144 (DEFAULT_TABLE table=default.Table_ddc8aae93ec243fd9f2aa7841c96bbca [id=505e004369be45e3950f15e43d47a14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.273905 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45421a6499f2468dadf9fb10866b0144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.277645 12820 tablet_service.cc:1467] Processing CreateTablet for tablet cb43ea3e28ba4d45a20f88f6fd4b3053 (DEFAULT_TABLE table=default.Table_9db7aa6cc91143a9be731a84793c16dc [id=58ca5d5422004bcd9950f0b0703b7695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.278012 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb43ea3e28ba4d45a20f88f6fd4b3053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.281581 12826 tablet_service.cc:1467] Processing CreateTablet for tablet eaccb511fba84ca0a7d9c7264490b6e0 (DEFAULT_TABLE table=default.Table_68bc366f32d440a9a0ddab8c2bcfb05e [id=52a4f1f2a0b84809a60715f6e5698b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.281911 12826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaccb511fba84ca0a7d9c7264490b6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.285669 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 33a082ee0898444ba9f0f5ee92ba89cc (DEFAULT_TABLE table=default.Table_946004ac61bd4421abf9cdfa64f35460 [id=1881dd13d96a4563a72c4791dcaccfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.286079 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a082ee0898444ba9f0f5ee92ba89cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.289629 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 6e080ee371c7405fb78faf0d77c56a9d (DEFAULT_TABLE table=default.Table_65858cfb30f343b5ae7629f09d0883b0 [id=bbe4d5d027284555846908c5b7b5eb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.290050 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e080ee371c7405fb78faf0d77c56a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.292984 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce0a5be54ee4bafae89d59d29b0c8df (DEFAULT_TABLE table=default.Table_3eff05dacdb8400894bde49d2337d7f3 [id=a332c12a7dca471eb9e79b7974ea26a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.293414 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce0a5be54ee4bafae89d59d29b0c8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.293902 14766 raft_consensus.cc:482] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.294037 14766 raft_consensus.cc:504] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.295202 14765 raft_consensus.cc:482] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.295354 14765 raft_consensus.cc:504] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.295727 12813 tablet_service.cc:1467] Processing CreateTablet for tablet b8d939ae47ca47beada51268bc94f446 (DEFAULT_TABLE table=default.Table_c05d8864f0ae4794a8a633bd250e7569 [id=cbff41d5d7a8485bb654d68d512fb589]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.296056 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d939ae47ca47beada51268bc94f446. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 19:56:58.296144 12781 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2b3268094cbf481698cb373c1bf6dfba
W20240506 19:56:58.297099 12921 consensus_peers.cc:485] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:58.297443 12781 leader_election.cc:341] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 796beb8e0a2c41d698bd4c05796263f1
I20240506 19:56:58.297646 14765 leader_election.cc:288] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:58.297865 12488 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 48943.8us
I20240506 19:56:58.300806 12816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fec440f0cc4490a71eb37d6a1155ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.306154 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce0a5be54ee4bafae89d59d29b0c8df (DEFAULT_TABLE table=default.Table_3eff05dacdb8400894bde49d2337d7f3 [id=a332c12a7dca471eb9e79b7974ea26a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.306635 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce0a5be54ee4bafae89d59d29b0c8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.306540 12954 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0cdb80cd54bcf99a5eaab85a13363 (DEFAULT_TABLE table=default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.307004 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0cdb80cd54bcf99a5eaab85a13363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.307057 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 45421a6499f2468dadf9fb10866b0144 (DEFAULT_TABLE table=default.Table_ddc8aae93ec243fd9f2aa7841c96bbca [id=505e004369be45e3950f15e43d47a14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.307248 12962 tablet_service.cc:1467] Processing CreateTablet for tablet cf5029e26a394caab1ea238c3924b8d9 (DEFAULT_TABLE table=default.Table_02edd5e110a34ea292b65d41b0963109 [id=a18b40d9fb2a4e348f06bdb0ef822aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.307397 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45421a6499f2468dadf9fb10866b0144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.307580 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5029e26a394caab1ea238c3924b8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.307608 12966 tablet_service.cc:1467] Processing CreateTablet for tablet fee2e352f0da4e9e80e858a123c984da (DEFAULT_TABLE table=default.Table_096d0d9806624f8dab6c61cfd67e5481 [id=8f3a327c7cff4778ace0d6588fbba7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.307976 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee2e352f0da4e9e80e858a123c984da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.309523 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 3411f2ae4a4e4e1b8609292f7ac86c70 (DEFAULT_TABLE table=default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.309885 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3411f2ae4a4e4e1b8609292f7ac86c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.313587 12956 tablet_service.cc:1467] Processing CreateTablet for tablet f501a21a1aa349118d47bdba73463816 (DEFAULT_TABLE table=default.Table_f162120d90f640e09da676cac7d0c187 [id=0b05c2baa470418c9f1c32a7de9167da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.313756 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 5cee27bc6e334dc3ae25f7ce5c40da00 (DEFAULT_TABLE table=default.Table_a0d74db25f05423f8b5ae3b82c7a54b1 [id=21294b97805743eb8444823b07a0ed7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.313969 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f501a21a1aa349118d47bdba73463816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.314116 12965 tablet_service.cc:1467] Processing CreateTablet for tablet b8d939ae47ca47beada51268bc94f446 (DEFAULT_TABLE table=default.Table_c05d8864f0ae4794a8a633bd250e7569 [id=cbff41d5d7a8485bb654d68d512fb589]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.314527 12964 tablet_service.cc:1467] Processing CreateTablet for tablet cb43ea3e28ba4d45a20f88f6fd4b3053 (DEFAULT_TABLE table=default.Table_9db7aa6cc91143a9be731a84793c16dc [id=58ca5d5422004bcd9950f0b0703b7695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.314904 12951 tablet_service.cc:1467] Processing CreateTablet for tablet eaccb511fba84ca0a7d9c7264490b6e0 (DEFAULT_TABLE table=default.Table_68bc366f32d440a9a0ddab8c2bcfb05e [id=52a4f1f2a0b84809a60715f6e5698b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.315223 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 33a082ee0898444ba9f0f5ee92ba89cc (DEFAULT_TABLE table=default.Table_946004ac61bd4421abf9cdfa64f35460 [id=1881dd13d96a4563a72c4791dcaccfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.315589 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 6e080ee371c7405fb78faf0d77c56a9d (DEFAULT_TABLE table=default.Table_65858cfb30f343b5ae7629f09d0883b0 [id=bbe4d5d027284555846908c5b7b5eb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.307410 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 32fec440f0cc4490a71eb37d6a1155ae (DEFAULT_TABLE table=default.Table_11ace0d42c9840d19108e3bfa5f1bb71 [id=29d018e8e77c4cbfa4cec78985b4de98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.315979 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3daa18d47842b089d2e94d29a3b21e (DEFAULT_TABLE table=default.Table_1590f38624f04f70a068b387db86e4bc [id=b21b01c7b4d146c5874a19d6621fb369]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.316319 12965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d939ae47ca47beada51268bc94f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.317359 14766 leader_election.cc:288] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.317507 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 5b472ba7729c4f6184e07eff618407f4 (DEFAULT_TABLE table=default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.317848 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b472ba7729c4f6184e07eff618407f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.321944 12964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb43ea3e28ba4d45a20f88f6fd4b3053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.322604 12843 raft_consensus.cc:3037] T 8a5217c257ff46e48c23563e496b6f08 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.325532 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaccb511fba84ca0a7d9c7264490b6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.328697 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a082ee0898444ba9f0f5ee92ba89cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.333595 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e080ee371c7405fb78faf0d77c56a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.337592 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fec440f0cc4490a71eb37d6a1155ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.341648 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f3daa18d47842b089d2e94d29a3b21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.345603 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee27bc6e334dc3ae25f7ce5c40da00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.351912 14739 consensus_queue.cc:1035] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:58.365126 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 45421a6499f2468dadf9fb10866b0144 (DEFAULT_TABLE table=default.Table_ddc8aae93ec243fd9f2aa7841c96bbca [id=505e004369be45e3950f15e43d47a14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.365463 13092 tablet_service.cc:1467] Processing CreateTablet for tablet cb43ea3e28ba4d45a20f88f6fd4b3053 (DEFAULT_TABLE table=default.Table_9db7aa6cc91143a9be731a84793c16dc [id=58ca5d5422004bcd9950f0b0703b7695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.365619 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45421a6499f2468dadf9fb10866b0144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.365775 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb43ea3e28ba4d45a20f88f6fd4b3053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.369596 13089 tablet_service.cc:1467] Processing CreateTablet for tablet eaccb511fba84ca0a7d9c7264490b6e0 (DEFAULT_TABLE table=default.Table_68bc366f32d440a9a0ddab8c2bcfb05e [id=52a4f1f2a0b84809a60715f6e5698b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.369947 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaccb511fba84ca0a7d9c7264490b6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.365464 13093 tablet_service.cc:1467] Processing CreateTablet for tablet 33a082ee0898444ba9f0f5ee92ba89cc (DEFAULT_TABLE table=default.Table_946004ac61bd4421abf9cdfa64f35460 [id=1881dd13d96a4563a72c4791dcaccfca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.374110 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a082ee0898444ba9f0f5ee92ba89cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.390167 14460 raft_consensus.cc:482] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.390323 14460 raft_consensus.cc:504] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.390960 14460 leader_election.cc:288] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:58.392210 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:58.392261 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:58.392767 12922 leader_election.cc:341] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:58.393028 12921 leader_election.cc:341] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.393139 12921 leader_election.cc:302] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.393618 14460 raft_consensus.cc:2726] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:58.394990 12515 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet bef5086682264414aedf3f1d50e88cc8
I20240506 19:56:58.396245 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_02ef9cd7cbba4ec1bc2971e5c8f43cda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.396927 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02ef9cd7cbba4ec1bc2971e5c8f43cda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.405985 13244 tablet_bootstrap.cc:492] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.408481 12516 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 7390b6d4e5054374bed2ca2b4e4bd74f
I20240506 19:56:58.409174 12516 catalog_manager.cc:6554] ReplaceTablet: tablet 7390b6d4e5054374bed2ca2b4e4bd74f deleted and replaced by tablet 4592c5bf88b74393ac0ebcf924d5f993
I20240506 19:56:58.409744 13244 ts_tablet_manager.cc:1367] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.156s user 0.006s sys 0.003s
I20240506 19:56:58.409665 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_36f124ac875b412fbe30315246e7c43c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.410445 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36f124ac875b412fbe30315246e7c43c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.410540 13244 raft_consensus.cc:348] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.410719 13244 raft_consensus.cc:374] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.410760 13244 raft_consensus.cc:729] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.416013 13244 consensus_queue.cc:260] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.416621 13244 ts_tablet_manager.cc:1398] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:58.416918 13244 tablet_bootstrap.cc:492] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.417425 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 007f9df7ce8e40a495a4eacb7a026b00 (DEFAULT_TABLE table=default.Table_a5d469d01eb3446b81546610731630c3 [id=dcf79d1fdf7b4750a4ed2fae4200783e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.417860 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f9df7ce8e40a495a4eacb7a026b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.417943 12948 tablet_service.cc:1467] Processing CreateTablet for tablet 6c89b80216d640ddbf65956fbd714fc3 (DEFAULT_TABLE table=default.Table_131e6a1d2bc240439badeffed52cd489 [id=2a02c78fdbd1463987715da7a840f0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.418313 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c89b80216d640ddbf65956fbd714fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.418761 12949 tablet_service.cc:1467] Processing CreateTablet for tablet ebda423ac5a54fd6894369f417018b4d (DEFAULT_TABLE table=default.Table_02b297f901384fd19bf275bcda8a2cab [id=ce3814e9575f446c9f0eec5e6b5e6d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.419092 12949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebda423ac5a54fd6894369f417018b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.421561 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 925b891e50af449099efd63a62a9fe8e (DEFAULT_TABLE table=default.Table_d73690e3683b43168f302fd778a0492b [id=6ecedb3d0a914866b8808b76813c1e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.421929 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925b891e50af449099efd63a62a9fe8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.424358 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 925b891e50af449099efd63a62a9fe8e (DEFAULT_TABLE table=default.Table_d73690e3683b43168f302fd778a0492b [id=6ecedb3d0a914866b8808b76813c1e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.424729 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925b891e50af449099efd63a62a9fe8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.425217 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 6c89b80216d640ddbf65956fbd714fc3 (DEFAULT_TABLE table=default.Table_131e6a1d2bc240439badeffed52cd489 [id=2a02c78fdbd1463987715da7a840f0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.425549 12967 tablet_service.cc:1467] Processing CreateTablet for tablet e01a75c227c44aaf8d4dd89fa6937335 (DEFAULT_TABLE table=default.Table_d8deac05c2a74b5583e5b0ddfd9ad894 [id=bedd1d32d0e844608dcb130ef224060e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.425901 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01a75c227c44aaf8d4dd89fa6937335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.426419 12813 tablet_service.cc:1467] Processing CreateTablet for tablet 1353e75a36d749fd8730744f88268742 (DEFAULT_TABLE table=default.Table_b86c2104b98d4723bfde3500950e640b [id=4d658c76e1e842588a67621ff1d945a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.426750 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1353e75a36d749fd8730744f88268742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.427799 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 1353e75a36d749fd8730744f88268742 (DEFAULT_TABLE table=default.Table_b86c2104b98d4723bfde3500950e640b [id=4d658c76e1e842588a67621ff1d945a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.428113 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1353e75a36d749fd8730744f88268742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.429520 12828 tablet_service.cc:1467] Processing CreateTablet for tablet a8d09ce439f54c41956f85f172643270 (DEFAULT_TABLE table=default.Table_12084251f64140ce8b5be5e06ed93ed3 [id=9d74bf3b46cf4be1a8bf414b00a003ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.429813 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d09ce439f54c41956f85f172643270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.432706 12818 tablet_service.cc:1467] Processing CreateTablet for tablet 829421d3782e4ea6a748a28609b78ffd (DEFAULT_TABLE table=default.Table_9e229895c2804c09bd30a74901fd77b6 [id=38db8971145547068978042310ba1d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.433040 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829421d3782e4ea6a748a28609b78ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.433521 12809 tablet_service.cc:1467] Processing CreateTablet for tablet 3450e9b8544d4accb79173d5166ded49 (DEFAULT_TABLE table=default.Table_0821a294cda34e98b1de703d3b5f8c1a [id=4d7b052cb45e4c1e9b33fdb0648f12d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.433665 12962 tablet_service.cc:1467] Processing CreateTablet for tablet a8d09ce439f54c41956f85f172643270 (DEFAULT_TABLE table=default.Table_12084251f64140ce8b5be5e06ed93ed3 [id=9d74bf3b46cf4be1a8bf414b00a003ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.433830 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3450e9b8544d4accb79173d5166ded49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.434006 12962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d09ce439f54c41956f85f172643270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.435989 12963 tablet_service.cc:1467] Processing CreateTablet for tablet 3450e9b8544d4accb79173d5166ded49 (DEFAULT_TABLE table=default.Table_0821a294cda34e98b1de703d3b5f8c1a [id=4d7b052cb45e4c1e9b33fdb0648f12d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240506 19:56:58.436192 12921 consensus_peers.cc:485] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.436374 12963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3450e9b8544d4accb79173d5166ded49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.436672 12821 tablet_service.cc:1467] Processing CreateTablet for tablet ebda423ac5a54fd6894369f417018b4d (DEFAULT_TABLE table=default.Table_02b297f901384fd19bf275bcda8a2cab [id=ce3814e9575f446c9f0eec5e6b5e6d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.436975 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebda423ac5a54fd6894369f417018b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.437839 13095 tablet_service.cc:1467] Processing CreateTablet for tablet 6e080ee371c7405fb78faf0d77c56a9d (DEFAULT_TABLE table=default.Table_65858cfb30f343b5ae7629f09d0883b0 [id=bbe4d5d027284555846908c5b7b5eb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.438228 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e080ee371c7405fb78faf0d77c56a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.439565 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 007f9df7ce8e40a495a4eacb7a026b00 (DEFAULT_TABLE table=default.Table_a5d469d01eb3446b81546610731630c3 [id=dcf79d1fdf7b4750a4ed2fae4200783e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.439873 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f9df7ce8e40a495a4eacb7a026b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.439952 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 66da88df0ab54047b31f871015acfde9 (DEFAULT_TABLE table=default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.440250 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66da88df0ab54047b31f871015acfde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.441840 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 007f9df7ce8e40a495a4eacb7a026b00 (DEFAULT_TABLE table=default.Table_a5d469d01eb3446b81546610731630c3 [id=dcf79d1fdf7b4750a4ed2fae4200783e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.442168 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007f9df7ce8e40a495a4eacb7a026b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.443177 12815 tablet_service.cc:1467] Processing CreateTablet for tablet e01a75c227c44aaf8d4dd89fa6937335 (DEFAULT_TABLE table=default.Table_d8deac05c2a74b5583e5b0ddfd9ad894 [id=bedd1d32d0e844608dcb130ef224060e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.443470 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01a75c227c44aaf8d4dd89fa6937335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.445151 12956 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9f8a418294952af9317052f1a1697 (DEFAULT_TABLE table=default.Table_c7551611bbef42be87218a270d30383c [id=756ca049e76744888fc42f22467dbb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.445540 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9f8a418294952af9317052f1a1697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.446120 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 925b891e50af449099efd63a62a9fe8e (DEFAULT_TABLE table=default.Table_d73690e3683b43168f302fd778a0492b [id=6ecedb3d0a914866b8808b76813c1e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.446444 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925b891e50af449099efd63a62a9fe8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.446668 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 1353e75a36d749fd8730744f88268742 (DEFAULT_TABLE table=default.Table_b86c2104b98d4723bfde3500950e640b [id=4d658c76e1e842588a67621ff1d945a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.447088 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1353e75a36d749fd8730744f88268742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.449545 13086 tablet_service.cc:1467] Processing CreateTablet for tablet e01a75c227c44aaf8d4dd89fa6937335 (DEFAULT_TABLE table=default.Table_d8deac05c2a74b5583e5b0ddfd9ad894 [id=bedd1d32d0e844608dcb130ef224060e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.449849 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01a75c227c44aaf8d4dd89fa6937335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.451297 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c89b80216d640ddbf65956fbd714fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.451929 12809 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9f8a418294952af9317052f1a1697 (DEFAULT_TABLE table=default.Table_c7551611bbef42be87218a270d30383c [id=756ca049e76744888fc42f22467dbb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.452234 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9f8a418294952af9317052f1a1697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.453524 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet f3e748d2c32a4b09abc872fed318e643 (table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]). Expected version 1 got 0
I20240506 19:56:58.453547 13100 tablet_service.cc:1467] Processing CreateTablet for tablet a8d09ce439f54c41956f85f172643270 (DEFAULT_TABLE table=default.Table_12084251f64140ce8b5be5e06ed93ed3 [id=9d74bf3b46cf4be1a8bf414b00a003ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.453877 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d09ce439f54c41956f85f172643270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.454383 12517 catalog_manager.cc:5484] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:58.454952 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 7b803ae289344f88b604a2053881d59b (table default.Table_6013611882be454da8ec1ea823261f6a [id=d2ffd5634f2042b7a85cbe74d6ca1df5]). Expected version 3 got 0
I20240506 19:56:58.455452 12517 catalog_manager.cc:5484] T 45da091c44824bd6bb8d390a5fb01ef8 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:58.460567 12514 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 4a49d621e94c4eadb63be52e58c3fab5
I20240506 19:56:58.461346 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f0624408bd8a4b41bccc73648b150210" } new_table_name: "default.Table_ef9202d673db4585a988cbb893c1eed0" modify_external_catalogs: true
I20240506 19:56:58.466742 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 3450e9b8544d4accb79173d5166ded49 (DEFAULT_TABLE table=default.Table_0821a294cda34e98b1de703d3b5f8c1a [id=4d7b052cb45e4c1e9b33fdb0648f12d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.467415 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3450e9b8544d4accb79173d5166ded49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.467795 13086 tablet_service.cc:1467] Processing CreateTablet for tablet ebda423ac5a54fd6894369f417018b4d (DEFAULT_TABLE table=default.Table_02b297f901384fd19bf275bcda8a2cab [id=ce3814e9575f446c9f0eec5e6b5e6d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.468120 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebda423ac5a54fd6894369f417018b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.470149 13094 tablet_service.cc:1467] Processing CreateTablet for tablet 66da88df0ab54047b31f871015acfde9 (DEFAULT_TABLE table=default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.470436 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66da88df0ab54047b31f871015acfde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.471113 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet bef5086682264414aedf3f1d50e88cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.471295 12821 tablet_service.cc:1514] Processing DeleteTablet for tablet 7390b6d4e5054374bed2ca2b4e4bd74f with delete_type TABLET_DATA_DELETED (replaced by tablet 4592c5bf88b74393ac0ebcf924d5f993) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.473480 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet bef5086682264414aedf3f1d50e88cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.473742 13100 tablet_service.cc:1514] Processing DeleteTablet for tablet 7390b6d4e5054374bed2ca2b4e4bd74f with delete_type TABLET_DATA_DELETED (replaced by tablet 4592c5bf88b74393ac0ebcf924d5f993) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.474150 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 66da88df0ab54047b31f871015acfde9 (DEFAULT_TABLE table=default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.474929 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66da88df0ab54047b31f871015acfde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.475106 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet bef5086682264414aedf3f1d50e88cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.476408 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 7390b6d4e5054374bed2ca2b4e4bd74f with delete_type TABLET_DATA_DELETED (replaced by tablet 4592c5bf88b74393ac0ebcf924d5f993) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.479539 13086 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9f8a418294952af9317052f1a1697 (DEFAULT_TABLE table=default.Table_c7551611bbef42be87218a270d30383c [id=756ca049e76744888fc42f22467dbb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.479979 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9f8a418294952af9317052f1a1697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.497368 14766 raft_consensus.cc:482] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.497536 14766 raft_consensus.cc:504] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.498147 14766 leader_election.cc:288] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:58.500211 12781 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 85bc7cfd006f4d4ca95541dab6873d32
I20240506 19:56:58.504426 13244 tablet_bootstrap.cc:654] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.543107 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 7300ca61d4474d2183f4281909400897 (DEFAULT_TABLE table=default.Table_111e3f9425804b4ca0bf599cc9377867 [id=8026f48a5250456a8cea07bb6446b845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.543777 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7300ca61d4474d2183f4281909400897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.545586 13244 tablet_bootstrap.cc:492] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.547375 13244 ts_tablet_manager.cc:1367] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.131s user 0.006s sys 0.000s
I20240506 19:56:58.548146 13244 raft_consensus.cc:348] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.548346 13244 raft_consensus.cc:374] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.548399 13244 raft_consensus.cc:729] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.548614 13244 consensus_queue.cc:260] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.548753 14757 raft_consensus.cc:482] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.548892 14757 raft_consensus.cc:504] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.549607 14757 leader_election.cc:288] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.550001 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f759b560e2ab41a6a076976ccca811d1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:58.550508 13058 leader_election.cc:341] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.551561 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f759b560e2ab41a6a076976ccca811d1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:58.551738 12987 raft_consensus.cc:2454] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:58.552209 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet bef5086682264414aedf3f1d50e88cc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bef5086682264414aedf3f1d50e88cc8 already in progress: creating tablet
W20240506 19:56:58.552613 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 7390b6d4e5054374bed2ca2b4e4bd74f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7390b6d4e5054374bed2ca2b4e4bd74f already in progress: creating tablet
W20240506 19:56:58.552759 12921 consensus_peers.cc:485] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.554028 13057 leader_election.cc:302] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.554663 14757 raft_consensus.cc:2781] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:58.554741 14757 raft_consensus.cc:482] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:58.554805 14757 raft_consensus.cc:3037] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.555511 13244 ts_tablet_manager.cc:1398] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240506 19:56:58.555786 13244 tablet_bootstrap.cc:492] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.555721 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_ff4a6a2744944a95aae0f535c062d338"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.556316 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff4a6a2744944a95aae0f535c062d338 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 19:56:58.558048 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet bef5086682264414aedf3f1d50e88cc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bef5086682264414aedf3f1d50e88cc8 already in progress: creating tablet
W20240506 19:56:58.558288 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 7390b6d4e5054374bed2ca2b4e4bd74f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7390b6d4e5054374bed2ca2b4e4bd74f already in progress: creating tablet
I20240506 19:56:58.560988 14757 raft_consensus.cc:504] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.561240 13244 tablet_bootstrap.cc:654] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:58.561887 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet bef5086682264414aedf3f1d50e88cc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bef5086682264414aedf3f1d50e88cc8 already in progress: creating tablet
I20240506 19:56:58.562209 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f759b560e2ab41a6a076976ccca811d1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:58.562764 13058 leader_election.cc:341] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.563012 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a49d621e94c4eadb63be52e58c3fab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.563395 14383 raft_consensus.cc:482] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.563577 14383 raft_consensus.cc:504] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.564205 14383 leader_election.cc:288] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:58.564766 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b011e94e2b47d7a51d6816576ea124" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:58.565430 12921 leader_election.cc:341] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.566043 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b011e94e2b47d7a51d6816576ea124" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:58.566231 13124 raft_consensus.cc:2454] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:58.566666 12922 leader_election.cc:302] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.567118 14383 raft_consensus.cc:2781] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:58.567189 14383 raft_consensus.cc:482] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:58.567242 14383 raft_consensus.cc:3037] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:58.568974 12783 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2b3268094cbf481698cb373c1bf6dfba
I20240506 19:56:58.569092 12783 leader_election.cc:302] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:58.569315 14766 raft_consensus.cc:2726] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:58.569690 12783 leader_election.cc:341] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 796beb8e0a2c41d698bd4c05796263f1
I20240506 19:56:58.569783 12783 leader_election.cc:302] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:58.569706 14383 raft_consensus.cc:504] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.569964 14766 raft_consensus.cc:2726] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:58.570487 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 7390b6d4e5054374bed2ca2b4e4bd74f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7390b6d4e5054374bed2ca2b4e4bd74f already in progress: creating tablet
I20240506 19:56:58.570962 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b011e94e2b47d7a51d6816576ea124" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:58.571144 13124 raft_consensus.cc:3037] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.573671 13124 raft_consensus.cc:2454] T e1b011e94e2b47d7a51d6816576ea124 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:58.574054 12922 leader_election.cc:302] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:58.574316 14460 raft_consensus.cc:2781] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:58.574417 14460 raft_consensus.cc:686] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:58.574656 14460 consensus_queue.cc:237] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.576932 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f759b560e2ab41a6a076976ccca811d1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:58.577189 12987 raft_consensus.cc:3037] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:58.577472 12506 catalog_manager.cc:4810] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): delete failed for tablet 4a49d621e94c4eadb63be52e58c3fab5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a49d621e94c4eadb63be52e58c3fab5 already in progress: creating tablet
I20240506 19:56:58.578294 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a49d621e94c4eadb63be52e58c3fab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.578648 14757 leader_election.cc:288] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.579396 13244 tablet_bootstrap.cc:492] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.579531 13244 ts_tablet_manager.cc:1367] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240506 19:56:58.579674 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b011e94e2b47d7a51d6816576ea124" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:58.580137 12921 leader_election.cc:341] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:58.580247 12514 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b1f0e4245774ccab0aa9c6c2ba49c75 consensus state has no leader
I20240506 19:56:58.580196 13244 raft_consensus.cc:348] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:58.580371 13244 raft_consensus.cc:374] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.580017 12987 raft_consensus.cc:2454] T f759b560e2ab41a6a076976ccca811d1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:58.580425 13244 raft_consensus.cc:729] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.580627 13244 consensus_queue.cc:260] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:58.580742 13057 leader_election.cc:302] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.581064 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_a2366efe03f84bdcabf1d6d62fc25ec7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:58.581444 14757 raft_consensus.cc:2781] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
W20240506 19:56:58.581277 12508 catalog_manager.cc:4810] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): delete failed for tablet 4a49d621e94c4eadb63be52e58c3fab5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a49d621e94c4eadb63be52e58c3fab5 already in progress: creating tablet
I20240506 19:56:58.582101 14757 raft_consensus.cc:686] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:58.582443 14757 consensus_queue.cc:237] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.580461 14383 leader_election.cc:288] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:58.583693 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2366efe03f84bdcabf1d6d62fc25ec7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.585711 13244 ts_tablet_manager.cc:1398] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:58.586104 13244 tablet_bootstrap.cc:492] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.589430 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 7300ca61d4474d2183f4281909400897 (DEFAULT_TABLE table=default.Table_111e3f9425804b4ca0bf599cc9377867 [id=8026f48a5250456a8cea07bb6446b845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.590036 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7300ca61d4474d2183f4281909400897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.591828 13244 tablet_bootstrap.cc:654] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.467368 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 6c89b80216d640ddbf65956fbd714fc3 (DEFAULT_TABLE table=default.Table_131e6a1d2bc240439badeffed52cd489 [id=2a02c78fdbd1463987715da7a840f0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.606382 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c89b80216d640ddbf65956fbd714fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.606696 13244 tablet_bootstrap.cc:492] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.606873 13244 ts_tablet_manager.cc:1367] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20240506 19:56:58.607323 14700 consensus_queue.cc:1035] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:58.607569 13244 raft_consensus.cc:348] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.607748 13244 raft_consensus.cc:374] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.607800 13244 raft_consensus.cc:729] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.458858 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 45da091c44824bd6bb8d390a5fb01ef8 (table default.Table_8afdb494805e4ba7838632b12912ec36 [id=22fff2da3af34564b794765f53773f81]). Expected version 2 got 0
I20240506 19:56:58.616009 12517 catalog_manager.cc:5484] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:58.617034 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]). Expected version 2 got 0
I20240506 19:56:58.617537 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 8a5217c257ff46e48c23563e496b6f08 (table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]). Expected version 1 got 0
I20240506 19:56:58.618368 12517 catalog_manager.cc:5484] T 6ace0ec43ba44060bcec2a50e1f48098 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:58.627369 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a49d621e94c4eadb63be52e58c3fab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.634349 14702 raft_consensus.cc:482] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.634547 14702 raft_consensus.cc:504] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.635353 14702 leader_election.cc:288] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:58.636502 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:58.636664 12783 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 85bc7cfd006f4d4ca95541dab6873d32
I20240506 19:56:58.636761 12783 leader_election.cc:302] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:58.637199 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:58.637425 13111 raft_consensus.cc:2454] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:58.637879 12922 leader_election.cc:302] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:58.638187 14766 raft_consensus.cc:2726] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:58.638366 12921 leader_election.cc:341] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.638689 14702 raft_consensus.cc:2781] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:58.638760 14702 raft_consensus.cc:482] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:58.638815 14702 raft_consensus.cc:3037] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.640517 13244 consensus_queue.cc:260] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.643164 13244 ts_tablet_manager.cc:1398] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240506 19:56:58.643182 14702 raft_consensus.cc:504] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.643455 13244 tablet_bootstrap.cc:492] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.644158 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:58.644333 13111 raft_consensus.cc:3037] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.647042 13111 raft_consensus.cc:2454] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:58.647503 12922 leader_election.cc:302] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:58.647785 14739 raft_consensus.cc:2781] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:58.655190 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:58.655750 12921 leader_election.cc:341] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.656762 13244 tablet_bootstrap.cc:654] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.656795 13091 tablet_service.cc:1467] Processing CreateTablet for tablet b8d939ae47ca47beada51268bc94f446 (DEFAULT_TABLE table=default.Table_c05d8864f0ae4794a8a633bd250e7569 [id=cbff41d5d7a8485bb654d68d512fb589]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 19:56:58.656912 14702 leader_election.cc:288] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:58.637378 12507 catalog_manager.cc:4810] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): delete failed for tablet 4a49d621e94c4eadb63be52e58c3fab5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a49d621e94c4eadb63be52e58c3fab5 already in progress: creating tablet
I20240506 19:56:58.645020 12515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_0fba0d9aab8e488b80a10c9afe3e7aee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.665076 12515 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fba0d9aab8e488b80a10c9afe3e7aee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.645745 12516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_7d6d951fd3fa48409ed1fadc66784c0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.667162 12516 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d6d951fd3fa48409ed1fadc66784c0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.674192 13244 tablet_bootstrap.cc:492] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.674348 13244 ts_tablet_manager.cc:1367] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.002s
I20240506 19:56:58.675069 13244 raft_consensus.cc:348] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.675297 13244 raft_consensus.cc:374] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.675343 13244 raft_consensus.cc:729] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.655896 12514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_faeb0284bba8473ebdd24f032788bdf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.677562 12514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faeb0284bba8473ebdd24f032788bdf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.681478 13244 consensus_queue.cc:260] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.671032 14656 raft_consensus.cc:482] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.682066 13244 ts_tablet_manager.cc:1398] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:56:58.682126 14656 raft_consensus.cc:504] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.682340 13244 tablet_bootstrap.cc:492] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.683002 14656 leader_election.cc:288] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.683750 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458314e4653748e48cf646c72373fc3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:58.683876 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458314e4653748e48cf646c72373fc3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:58.684118 12987 raft_consensus.cc:2454] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:58.684247 13058 leader_election.cc:341] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.684844 13057 leader_election.cc:302] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.685282 14656 raft_consensus.cc:2781] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:58.685372 14656 raft_consensus.cc:482] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:58.685431 14656 raft_consensus.cc:3037] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.688026 14656 raft_consensus.cc:504] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.689060 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458314e4653748e48cf646c72373fc3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:58.689543 13058 leader_election.cc:341] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.689918 14656 leader_election.cc:288] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.690243 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458314e4653748e48cf646c72373fc3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:58.690454 12987 raft_consensus.cc:3037] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.694188 12987 raft_consensus.cc:2454] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:58.695446 13057 leader_election.cc:302] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.696116 14656 raft_consensus.cc:2781] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:58.699517 13244 tablet_bootstrap.cc:654] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.699782 14782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:58.700150 14782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:58.701758 12531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "190a2eabb30d4154960a44eb50a8a87d" is_pre_election: true
I20240506 19:56:58.705586 14782 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:56:58.712580 13498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
I20240506 19:56:58.713610 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:56:58.714319 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6ace0ec43ba44060bcec2a50e1f48098 (table default.Table_61becbd043ee470ca2ee3f4969d59971 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]). Expected version 2 got 0
I20240506 19:56:58.715056 12517 catalog_manager.cc:5484] T 838d9e7986164ffd916d2d842dc61f6e P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:58.715801 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 838d9e7986164ffd916d2d842dc61f6e (table default.Table_3270a7fb14b445ccbe0524f7b444e06d [id=f424bbb43b2a47449f998c21dae05cb9]). Expected version 3 got 0
I20240506 19:56:58.716320 12517 catalog_manager.cc:5576] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527) does not have the latest schema for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (table default.Table_23747e8c1b314e4eaa4566b557389261 [id=a8b5f55150a54dcf8a19dc9e1153ea87]). Expected version 1 got 0
I20240506 19:56:58.716617 12517 catalog_manager.cc:5484] T 10ca9a1eaffc45f486d7a7e937a3e787 P 059a5c4a3f7e49ef95af4b8341702745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194). New cstate: current_term: 1 leader_uuid: "059a5c4a3f7e49ef95af4b8341702745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } health_report { overall_health: HEALTHY } } }
I20240506 19:56:58.717311 14782 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:56:58.717489 12513 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2] alter complete (version 2)
I20240506 19:56:58.717547 12521 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 304a1401627948bbaff78fae4d42ab22
W20240506 19:56:58.718560 12513 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.195:39651 (request call id 171) took 2142 ms (client timeout 1000 ms). Trace:
I20240506 19:56:58.718474 12521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_2da343363b4b4d9a9d82f1ab83c1c392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.719019 12521 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2da343363b4b4d9a9d82f1ab83c1c392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.719594 13093 tablet_service.cc:1514] Processing DeleteTablet for tablet 304a1401627948bbaff78fae4d42ab22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.719822 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ef5a2799b447f992207452dbe39645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:44 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.719900 13246 tablet_replica.cc:330] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:58.720129 13246 raft_consensus.cc:2229] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.720309 13246 raft_consensus.cc:2258] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.720517 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 1750c4dd8cdc40509c553b63d166c466 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:46 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.720777 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet ddae4b0d8d0943d99f54904266f293d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.720890 13246 ts_tablet_manager.cc:1875] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:58.721457 14779 tablet.cc:1722] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:58.723613 12963 tablet_service.cc:1514] Processing DeleteTablet for tablet 458314e4653748e48cf646c72373fc3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:41 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.723848 14557 tablet_replica.cc:330] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:58.724042 14557 raft_consensus.cc:2229] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.724174 14557 raft_consensus.cc:2258] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240506 19:56:58.718645 12513 rpcz_store.cc:269] 0506 19:56:56.575697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.697719 (+122022us) service_pool.cc:225] Handling call
0506 19:56:58.718538 (+2020819us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:57.194325 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:57.194433 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:57.194441 (+ 8us) write_op.cc:435] Acquired schema lock
0506 19:56:57.194443 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:57.194550 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:57.194562 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:57.194568 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:57.194637 (+ 69us) tablet.cc:699] Row locks acquired
0506 19:56:57.194639 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:57.194671 (+ 32us) write_op.cc:270] Start()
0506 19:56:57.194706 (+ 35us) write_op.cc:276] Timestamp: P: 1715025417194711 usec, L: 0
0506 19:56:57.194710 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:57.194905 (+ 195us) log.cc:842] Serialized 356 byte log entry
0506 19:56:57.310963 (+116058us) op_driver.cc:444] REPLICATION: finished
0506 19:56:57.311784 (+ 821us) write_op.cc:301] APPLY: starting
0506 19:56:57.311796 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:57.311829 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:57.311856 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:57.311894 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:57.311920 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:57.311927 (+ 7us) write_op.cc:312] APPLY: finished
0506 19:56:57.311981 (+ 54us) log.cc:842] Serialized 23 byte log entry
0506 19:56:57.312020 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.312033 (+ 13us) write_op.cc:454] Released schema lock
0506 19:56:57.312038 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:57.312082 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.312086 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 19:56:57.365853 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:57.365938 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:57.365944 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:57.365946 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:57.366028 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:57.366037 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:57.366041 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:57.366086 (+ 45us) tablet.cc:699] Row locks acquired
0506 19:56:57.366089 (+ 3us) write_op.cc:260] PREPARE: finished
0506 19:56:57.366113 (+ 24us) write_op.cc:270] Start()
0506 19:56:57.366136 (+ 23us) write_op.cc:276] Timestamp: P: 1715025417366222 usec, L: 0
0506 19:56:57.366140 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 19:56:57.366269 (+ 129us) log.cc:842] Serialized 356 byte log entry
0506 19:56:57.708275 (+342006us) op_driver.cc:444] REPLICATION: finished
0506 19:56:57.709666 (+ 1391us) write_op.cc:301] APPLY: starting
0506 19:56:57.709675 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:57.709701 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:57.709721 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:57.709753 (+ 32us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:57.709775 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:57.709782 (+ 7us) write_op.cc:312] APPLY: finished
0506 19:56:57.709830 (+ 48us) log.cc:842] Serialized 23 byte log entry
0506 19:56:57.709872 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.709883 (+ 11us) write_op.cc:454] Released schema lock
0506 19:56:57.709888 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:57.709930 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.709933 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 19:56:57.723471 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:57.723567 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:57.723573 (+ 6us) write_op.cc:435] Acquired schema lock
0506 19:56:57.723575 (+ 2us) tablet.cc:660] Decoding operations
0506 19:56:57.723679 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:57.723688 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:57.723692 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 19:56:57.723740 (+ 48us) tablet.cc:699] Row locks acquired
0506 19:56:57.723742 (+ 2us) write_op.cc:260] PREPARE: finished
0506 19:56:57.723773 (+ 31us) write_op.cc:270] Start()
0506 19:56:57.723800 (+ 27us) write_op.cc:276] Timestamp: P: 1715025417723831 usec, L: 0
0506 19:56:57.723803 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 19:56:57.723932 (+ 129us) log.cc:842] Serialized 356 byte log entry
0506 19:56:57.833106 (+109174us) op_driver.cc:444] REPLICATION: finished
0506 19:56:57.833767 (+ 661us) write_op.cc:301] APPLY: starting
0506 19:56:57.833777 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:57.833804 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:57.833825 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:57.833866 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:57.833891 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0506 19:56:57.833898 (+ 7us) write_op.cc:312] APPLY: finished
0506 19:56:57.833943 (+ 45us) log.cc:842] Serialized 23 byte log entry
0506 19:56:57.833976 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.833988 (+ 12us) write_op.cc:454] Released schema lock
0506 19:56:57.833992 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:57.834029 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.834033 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":788,"apply.run_cpu_time_us":342,"apply.run_wall_time_us":341,"num_ops":1,"prepare.queue_time_us":198179,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":673,"raft.queue_time_us":991,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":116236}],["op",{"apply.queue_time_us":1365,"apply.run_cpu_time_us":304,"apply.run_wall_time_us":304,"mutex_wait_us":42800,"num_ops":1,"prepare.queue_time_us":48184,"prepare.run_cpu_time_us":459,"prepare.run_wall_time_us":457,"replication_time_us":342122}],["op",{"apply.queue_time_us":629,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":300,"num_ops":1,"prepare.queue_time_us":4513,"prepare.run_cpu_time_us":513,"prepare.run_wall_time_us":510,"replication_time_us":109288}]]}
I20240506 19:56:58.724666 14557 ts_tablet_manager.cc:1875] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:58.724651 12513 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_807ff2b24da64f7396c8dc15bdc46c88" } modify_external_catalogs: true
I20240506 19:56:58.724810 12513 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_807ff2b24da64f7396c8dc15bdc46c88" } modify_external_catalogs: true
W20240506 19:56:58.725970 12522 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.194:59149 (request call id 138) took 1960 ms (client timeout 1000 ms). Trace:
W20240506 19:56:58.726050 12522 rpcz_store.cc:269] 0506 19:56:56.765595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:56.765649 (+ 54us) service_pool.cc:225] Handling call
0506 19:56:58.725934 (+1960285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 19:56:57.230040 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0506 19:56:57.233250 (+ 3210us) write_op.cc:432] Acquiring schema lock in shared mode
0506 19:56:57.233257 (+ 7us) write_op.cc:435] Acquired schema lock
0506 19:56:57.233260 (+ 3us) tablet.cc:660] Decoding operations
0506 19:56:57.236838 (+ 3578us) write_op.cc:620] Acquiring the partition lock for write op
0506 19:56:57.236853 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 19:56:57.236859 (+ 6us) tablet.cc:683] Acquiring locks for 3 operations
0506 19:56:57.242056 (+ 5197us) tablet.cc:699] Row locks acquired
0506 19:56:57.242079 (+ 23us) write_op.cc:260] PREPARE: finished
0506 19:56:57.242138 (+ 59us) write_op.cc:270] Start()
0506 19:56:57.242183 (+ 45us) write_op.cc:276] Timestamp: P: 1715025417242189 usec, L: 0
0506 19:56:57.242189 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 19:56:57.242431 (+ 242us) log.cc:842] Serialized 1293 byte log entry
0506 19:56:57.330298 (+ 87867us) op_driver.cc:444] REPLICATION: finished
0506 19:56:57.331940 (+ 1642us) write_op.cc:301] APPLY: starting
0506 19:56:57.331948 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0506 19:56:57.331978 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0506 19:56:57.331997 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 19:56:57.332050 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 19:56:57.332072 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0506 19:56:57.332078 (+ 6us) write_op.cc:312] APPLY: finished
0506 19:56:57.332124 (+ 46us) log.cc:842] Serialized 35 byte log entry
0506 19:56:57.332156 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.332169 (+ 13us) write_op.cc:454] Released schema lock
0506 19:56:57.332173 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0506 19:56:57.332212 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0506 19:56:57.332215 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":1619,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":309,"num_ops":3,"prepare.queue_time_us":592,"prepare.run_cpu_time_us":1044,"prepare.run_wall_time_us":12968,"replication_time_us":88097}]]}
I20240506 19:56:58.726773 12522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_22d4c2a939a34f7c9a0bbd4eb808d058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 19:56:58.727260 12522 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d4c2a939a34f7c9a0bbd4eb808d058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.727375 13246 ts_tablet_manager.cc:1888] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:58.727479 13246 log.cc:1196] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/304a1401627948bbaff78fae4d42ab22
I20240506 19:56:58.727828 13246 ts_tablet_manager.cc:1909] T 304a1401627948bbaff78fae4d42ab22 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
W20240506 19:56:58.728664 12518 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.193:41395 (request call id 188) took 938 ms (client timeout 1000 ms). Trace:
W20240506 19:56:58.728749 12518 rpcz_store.cc:269] 0506 19:56:57.790191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:57.821513 (+ 31322us) service_pool.cc:225] Handling call
0506 19:56:58.728555 (+907042us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x60f00002ee78. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 000000000095519a 00007f8fe01ff19d 00007f8fe01fedbe 00007f8ff36b862d 00007f8ff36b85cc 00007f8fe1377283 00007f8ff37f109b 00007f8fea7e2786 00007f8fea7e24b5 00007f8fe6f1104e 00007f8fe6f0ff2e
0506 19:56:58.728629 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":13961472}
I20240506 19:56:58.728854 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 304a1401627948bbaff78fae4d42ab22 (table default.Table_322cb621e5734d8bb9694e30ac509ea7 [id=3a715f69316d4a67b261118ff0069c11]) successfully deleted
I20240506 19:56:58.729141 14739 raft_consensus.cc:686] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:58.729247 13246 tablet_replica.cc:330] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:58.729156 12518 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33592:
name: "default.Table_922f89e002cf480a9fca0e23f0eeb801"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 19:56:58.729454 13246 raft_consensus.cc:2229] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
W20240506 19:56:58.729652 12518 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_922f89e002cf480a9fca0e23f0eeb801 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 19:56:58.729794 13246 raft_consensus.cc:2258] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.730316 13246 ts_tablet_manager.cc:1875] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:58.730907 12520 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.12.48.195:39651 (request call id 172) took 1149 ms (client timeout 1000 ms). Trace:
W20240506 19:56:58.730988 12520 rpcz_store.cc:269] 0506 19:56:57.581250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:57.712735 (+131485us) service_pool.cc:225] Handling call
0506 19:56:58.730817 (+1018082us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x60f00002ee78. stack: 00007f8fe0209756 00007f8fe0209510 00007f8fdf23c48d 0000000000955804 00000000009557cd 000000000095519a 00007f8fe01ff19d 00007f8fe01fedbe 00007f8ff36b862d 00007f8ff36b85cc 00007f8fe1377283 00007f8ff37f109b 00007f8fea7e2786 00007f8fea7e24b5 00007f8fe6f1104e 00007f8fe6f0ff2e
0506 19:56:58.730891 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":17,"spinlock_wait_cycles":16732032}
I20240506 19:56:58.731222 12520 master_service.cc:946] ReplaceTablet: received request to replace tablet 98571fa2168644969c851a8edd343a7a from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:58.733407 14557 ts_tablet_manager.cc:1888] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:58.733587 14557 log.cc:1196] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/458314e4653748e48cf646c72373fc3e
I20240506 19:56:58.734037 14557 ts_tablet_manager.cc:1909] T 458314e4653748e48cf646c72373fc3e P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:58.734277 12966 tablet_service.cc:1514] Processing DeleteTablet for tablet 260d7ab458794df496012cb80466f59c with delete_type TABLET_DATA_DELETED (replaced by tablet f1809a8a6a9f458bb5d9de9bcf7bd179) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.735517 12965 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ff7a3638004012b08c973192c055a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:40 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.735798 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.736977 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f382a89f94e4fdf9b8266d049635329 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:48 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.737236 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet bff7050bbc2e47cf823d25ec10b89718 with delete_type TABLET_DATA_DELETED (replaced by tablet cc22261ab2034e818834ba97c2760dad) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.737530 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 304a1401627948bbaff78fae4d42ab22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.737824 13090 tablet_service.cc:1514] Processing DeleteTablet for tablet 880ecc75bab446e39c687f3c8b740361 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4a9693c9fb411db07847fede883920) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.733474 14739 consensus_queue.cc:237] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.739356 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b9731a473449b19a9e95653a9d0830 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:43 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.739679 12506 catalog_manager.cc:4830] TS 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): tablet 458314e4653748e48cf646c72373fc3e (table default.Table_3ce0a12d16814dc9a767895fe39f1247 [id=55b6d1ff222b47a7831fce79b949314c]) successfully deleted
I20240506 19:56:58.741029 12816 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0f74c7c4cd42d293fa244de3fd693b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:56 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.741410 13242 tablet_replica.cc:330] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:58.741645 13242 raft_consensus.cc:2229] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.741822 13242 raft_consensus.cc:2258] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.742455 13242 ts_tablet_manager.cc:1875] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:58.749476 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 54408.5us
I20240506 19:56:58.749576 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 796beb8e0a2c41d698bd4c05796263f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:45 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.751009 13242 ts_tablet_manager.cc:1888] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:58.751211 13242 log.cc:1196] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/ad0f74c7c4cd42d293fa244de3fd693b
I20240506 19:56:58.751590 13242 ts_tablet_manager.cc:1909] T ad0f74c7c4cd42d293fa244de3fd693b P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:58.752254 13242 tablet_replica.cc:330] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:58.752559 13242 raft_consensus.cc:2229] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.752784 13242 raft_consensus.cc:2258] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.753517 13242 ts_tablet_manager.cc:1875] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:58.754812 13246 ts_tablet_manager.cc:1888] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:58.754973 13246 log.cc:1196] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/50ef5a2799b447f992207452dbe39645
I20240506 19:56:58.755268 13246 ts_tablet_manager.cc:1909] T 50ef5a2799b447f992207452dbe39645 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:58.755223 12507 catalog_manager.cc:4830] TS ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): tablet ad0f74c7c4cd42d293fa244de3fd693b (table default.Table_f2e21527b828489cab8c6dc854d0f813 [id=fb03660d814a4c62b27f9637313a7869]) successfully deleted
I20240506 19:56:58.755271 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 304a1401627948bbaff78fae4d42ab22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.755632 14557 tablet_replica.cc:330] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:58.755934 14557 raft_consensus.cc:2229] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.756145 14557 raft_consensus.cc:2258] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.756731 14557 ts_tablet_manager.cc:1875] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:58.758107 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 50ef5a2799b447f992207452dbe39645 (table default.Table_76271c0b874e41b29c550ad9d849d913 [id=9d0bea602ade4b3cb1ae8d46915d4ae3]) successfully deleted
I20240506 19:56:58.758747 13246 tablet_replica.cc:330] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:58.759001 13246 raft_consensus.cc:2229] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.760927 13091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d939ae47ca47beada51268bc94f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 19:56:58.759169 13246 raft_consensus.cc:2258] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.764652 13246 ts_tablet_manager.cc:1875] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:58.783075 13244 tablet_bootstrap.cc:492] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.783243 13244 ts_tablet_manager.cc:1367] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.010s
I20240506 19:56:58.784037 12515 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d8deac05c2a74b5583e5b0ddfd9ad894" } new_table_name: "default.Table_f09a4c6b266742568b795b84007e3bf0" modify_external_catalogs: true
I20240506 19:56:58.789433 13244 raft_consensus.cc:348] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.789646 13244 raft_consensus.cc:374] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.789711 13244 raft_consensus.cc:729] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.787930 14784 raft_consensus.cc:482] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.790380 14784 raft_consensus.cc:504] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.791776 14784 leader_election.cc:288] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:58.792651 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3049a9fa8049e5b84230e5e6e8245c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:58.793371 12922 leader_election.cc:341] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.794209 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3049a9fa8049e5b84230e5e6e8245c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:58.794462 14656 raft_consensus.cc:686] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:58.794791 14656 consensus_queue.cc:237] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:58.795567 12921 leader_election.cc:341] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.795763 12921 leader_election.cc:302] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:58.796092 14739 raft_consensus.cc:2726] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:58.811417 13246 ts_tablet_manager.cc:1888] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:58.811576 13246 log.cc:1196] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/1750c4dd8cdc40509c553b63d166c466
I20240506 19:56:58.811908 13246 ts_tablet_manager.cc:1909] T 1750c4dd8cdc40509c553b63d166c466 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:58.812498 12508 catalog_manager.cc:4830] TS 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): tablet 1750c4dd8cdc40509c553b63d166c466 (table default.Table_62cc668dd6f64dbcbe27b1bcfdd9d410 [id=1f5d46e4e6cb45e0ad944590ca94dd5e]) successfully deleted
I20240506 19:56:58.819923 12519 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 20aa06290b9a4814a006c423ccd8fc46
W20240506 19:56:58.821337 13245 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.042s sys 0.042s
W20240506 19:56:58.821430 13245 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.555s user 0.042s sys 0.042s
I20240506 19:56:58.823750 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet 20aa06290b9a4814a006c423ccd8fc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.823840 13245 tablet_bootstrap.cc:492] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.824156 13245 ts_tablet_manager.cc:1367] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.576s user 0.045s sys 0.042s
I20240506 19:56:58.824968 13245 raft_consensus.cc:348] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.825953 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet 20aa06290b9a4814a006c423ccd8fc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.829355 13245 raft_consensus.cc:374] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.829481 13245 raft_consensus.cc:729] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.829829 13245 consensus_queue.cc:260] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.833762 12513 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet e0fbbc77c0b64c2fa3d125e074eecd87
I20240506 19:56:58.835377 12522 master_service.cc:946] ReplaceTablet: received request to replace tablet c5751f7370d142a2914ac75b233e8c56 from {username='slave'} at 127.0.0.1:33592
I20240506 19:56:58.835427 13245 ts_tablet_manager.cc:1398] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240506 19:56:58.835734 12514 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_f9f5211355fa4a10b23a184e7c55ae7c" } new_table_name: "default.Table_5ac321da698a4689a0e094a882919160" modify_external_catalogs: true
I20240506 19:56:58.836542 12513 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_d2889007e246431bb796c67ca72db0aa" } new_table_name: "default.Table_8785df10c281478e8d3f15bf24c106ea" modify_external_catalogs: true
I20240506 19:56:58.836555 12518 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e4fb89d4dd964179b1cd1c2f2c91567c" } modify_external_catalogs: true
I20240506 19:56:58.836726 12518 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_e4fb89d4dd964179b1cd1c2f2c91567c" } modify_external_catalogs: true
I20240506 19:56:58.837932 12521 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_66fa4fdbd72742d9aa51e8911f3b72cc" } new_table_name: "default.Table_c7f2dabbbafa4177a30f23cb062955ce" modify_external_catalogs: true
I20240506 19:56:58.840032 12520 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 190a2eabb30d4154960a44eb50a8a87d: Sending DeleteTablet for 3 replicas of tablet 98571fa2168644969c851a8edd343a7a
I20240506 19:56:58.840504 12520 catalog_manager.cc:6554] ReplaceTablet: tablet 98571fa2168644969c851a8edd343a7a deleted and replaced by tablet 70e43e0a20cf4f05a7bc69278a83d1e3
I20240506 19:56:58.840744 12520 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_114d9e27c6d74425802c46e8610d3343" } modify_external_catalogs: true
I20240506 19:56:58.840878 12520 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33592:
table { table_name: "default.Table_114d9e27c6d74425802c46e8610d3343" } modify_external_catalogs: true
I20240506 19:56:58.841418 12483 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu pid 12491
I20240506 19:56:58.841967 13246 tablet_replica.cc:330] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:58.842133 13245 tablet_bootstrap.cc:492] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:58.843607 13245 tablet_bootstrap.cc:654] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.842145 13246 raft_consensus.cc:2229] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:58.844238 13246 raft_consensus.cc:2258] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:58.844754 13246 ts_tablet_manager.cc:1875] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:58.844892 12552 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 19:56:58.846942 14766 raft_consensus.cc:482] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.847165 14766 raft_consensus.cc:504] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.847942 14766 leader_election.cc:288] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.848404 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 98571fa2168644969c851a8edd343a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 70e43e0a20cf4f05a7bc69278a83d1e3) from {username='slave'} at 127.0.0.1:50504
I20240506 19:56:58.848829 12818 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fbbc77c0b64c2fa3d125e074eecd87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:50504
W20240506 19:56:58.851758 12783 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:56:58.852952 13094 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fbbc77c0b64c2fa3d125e074eecd87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:56.607334 14160 rpcz_store.cc:276] 0506 19:56:49.467531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 19:56:49.469635 (+ 2104us) service_pool.cc:225] Handling call
0506 19:56:53.829230 (+4359595us) raft_consensus.cc:1457] Updating replica for 415 ops
0506 19:56:54.015007 (+185777us) raft_consensus.cc:1520] Early marking committed up to index 584
0506 19:56:54.209574 (+194567us) raft_consensus.cc:1525] Triggering prepare for 415 ops
0506 19:56:54.222594 (+ 13020us) write_op.cc:270] Start()
0506 19:56:54.222704 (+ 110us) write_op.cc:276] Timestamp: P: 1715025402233819 usec, L: 0
0506 19:56:54.233612 (+ 10908us) write_op.cc:270] Start()
0506 19:56:54.233646 (+ 34us) write_op.cc:276] Timestamp: P: 1715025402250202 usec, L: 0
0506 19:56:54.233836 (+ 190us) write_op.cc:270] Start()
0506 19:56:54.233856 (+ 20us) write_op.cc:276] Timestamp: P: 1715025402321631 usec, L: 0
0506 19:56:54.233976 (+ 120us) write_op.cc:270] Start()
0506 19:56:54.233992 (+ 16us) write_op.cc:276] Timestamp: P: 1715025402327235 usec, L: 0
0506 19:56:54.234121 (+ 129us) write_op.cc:270] Start()
0506 19:56:54.234138 (+ 17us) write_op.cc:276] Timestamp: P: 1715025402352649 usec, L: 0
0506 19:56:54.234315 (+ 177us) write_op.cc:270] Start()
0506 19:56:54.234334 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402395789 usec, L: 0
0506 19:56:54.234452 (+ 118us) write_op.cc:270] Start()
0506 19:56:54.234503 (+ 51us) write_op.cc:276] Timestamp: P: 1715025402457347 usec, L: 0
0506 19:56:54.234655 (+ 152us) write_op.cc:270] Start()
0506 19:56:54.234674 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402513827 usec, L: 0
0506 19:56:54.234825 (+ 151us) write_op.cc:270] Start()
0506 19:56:54.234844 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402535295 usec, L: 0
0506 19:56:54.234945 (+ 101us) write_op.cc:270] Start()
0506 19:56:54.234960 (+ 15us) write_op.cc:276] Timestamp: P: 1715025402554053 usec, L: 0
0506 19:56:54.235072 (+ 112us) write_op.cc:270] Start()
0506 19:56:54.235089 (+ 17us) write_op.cc:276] Timestamp: P: 1715025402569663 usec, L: 0
0506 19:56:54.235202 (+ 113us) write_op.cc:270] Start()
0506 19:56:54.235219 (+ 17us) write_op.cc:276] Timestamp: P: 1715025402578566 usec, L: 0
0506 19:56:54.235395 (+ 176us) write_op.cc:270] Start()
0506 19:56:54.235414 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402581780 usec, L: 0
0506 19:56:54.235571 (+ 157us) write_op.cc:270] Start()
0506 19:56:54.235594 (+ 23us) write_op.cc:276] Timestamp: P: 1715025402614054 usec, L: 0
0506 19:56:54.235730 (+ 136us) write_op.cc:270] Start()
0506 19:56:54.235753 (+ 23us) write_op.cc:276] Timestamp: P: 1715025402615038 usec, L: 0
0506 19:56:54.235897 (+ 144us) write_op.cc:270] Start()
0506 19:56:54.235918 (+ 21us) write_op.cc:276] Timestamp: P: 1715025402615590 usec, L: 0
0506 19:56:54.236069 (+ 151us) write_op.cc:270] Start()
0506 19:56:54.236101 (+ 32us) write_op.cc:276] Timestamp: P: 1715025402616122 usec, L: 0
0506 19:56:54.236232 (+ 131us) write_op.cc:270] Start()
0506 19:56:54.236252 (+ 20us) write_op.cc:276] Timestamp: P: 1715025402616673 usec, L: 0
0506 19:56:54.236650 (+ 398us) write_op.cc:270] Start()
0506 19:56:54.236682 (+ 32us) write_op.cc:276] Timestamp: P: 1715025402617247 usec, L: 0
0506 19:56:54.236805 (+ 123us) write_op.cc:270] Start()
0506 19:56:54.236823 (+ 18us) write_op.cc:276] Timestamp: P: 1715025402619838 usec, L: 0
0506 19:56:54.236937 (+ 114us) write_op.cc:270] Start()
0506 19:56:54.236954 (+ 17us) write_op.cc:276] Timestamp: P: 1715025402620639 usec, L: 0
0506 19:56:54.237097 (+ 143us) write_op.cc:270] Start()
0506 19:56:54.237116 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402661847 usec, L: 0
0506 19:56:54.237285 (+ 169us) write_op.cc:270] Start()
0506 19:56:54.237336 (+ 51us) write_op.cc:276] Timestamp: P: 1715025402665407 usec, L: 0
0506 19:56:54.237532 (+ 196us) write_op.cc:270] Start()
0506 19:56:54.237555 (+ 23us) write_op.cc:276] Timestamp: P: 1715025402670491 usec, L: 0
0506 19:56:54.237729 (+ 174us) write_op.cc:270] Start()
0506 19:56:54.237753 (+ 24us) write_op.cc:276] Timestamp: P: 1715025402672661 usec, L: 0
0506 19:56:54.237873 (+ 120us) write_op.cc:270] Start()
0506 19:56:54.237890 (+ 17us) write_op.cc:276] Timestamp: P: 1715025402698750 usec, L: 0
0506 19:56:54.238002 (+ 112us) write_op.cc:270] Start()
0506 19:56:54.238017 (+ 15us) write_op.cc:276] Timestamp: P: 1715025402703068 usec, L: 0
0506 19:56:54.238133 (+ 116us) write_op.cc:270] Start()
0506 19:56:54.238151 (+ 18us) write_op.cc:276] Timestamp: P: 1715025402739617 usec, L: 0
0506 19:56:54.238245 (+ 94us) write_op.cc:270] Start()
0506 19:56:54.238260 (+ 15us) write_op.cc:276] Timestamp: P: 1715025402743536 usec, L: 0
0506 19:56:54.239594 (+ 1334us) write_op.cc:270] Start()
0506 19:56:54.239618 (+ 24us) write_op.cc:276] Timestamp: P: 1715025402769311 usec, L: 0
0506 19:56:54.239740 (+ 122us) write_op.cc:270] Start()
0506 19:56:54.239756 (+ 16us) write_op.cc:276] Timestamp: P: 1715025402771627 usec, L: 0
0506 19:56:54.239853 (+ 97us) write_op.cc:270] Start()
0506 19:56:54.239868 (+ 15us) write_op.cc:276] Timestamp: P: 1715025402791574 usec, L: 0
0506 19:56:54.245613 (+ 5745us) write_op.cc:270] Start()
0506 19:56:54.245716 (+ 103us) write_op.cc:276] Timestamp: P: 1715025402799786 usec, L: 0
0506 19:56:54.246028 (+ 312us) write_op.cc:270] Start()
0506 19:56:54.246065 (+ 37us) write_op.cc:276] Timestamp: P: 1715025402803799 usec, L: 0
0506 19:56:54.246215 (+ 150us) write_op.cc:270] Start()
0506 19:56:54.246237 (+ 22us) write_op.cc:276] Timestamp: P: 1715025402824276 usec, L: 0
0506 19:56:54.246411 (+ 174us) write_op.cc:270] Start()
0506 19:56:54.246430 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402832172 usec, L: 0
0506 19:56:54.246528 (+ 98us) write_op.cc:270] Start()
0506 19:56:54.246560 (+ 32us) write_op.cc:276] Timestamp: P: 1715025402861403 usec, L: 0
0506 19:56:54.246702 (+ 142us) write_op.cc:270] Start()
0506 19:56:54.246782 (+ 80us) write_op.cc:276] Timestamp: P: 1715025402876835 usec, L: 0
0506 19:56:54.246993 (+ 211us) write_op.cc:270] Start()
0506 19:56:54.247014 (+ 21us) write_op.cc:276] Timestamp: P: 1715025402881863 usec, L: 0
0506 19:56:54.247830 (+ 816us) write_op.cc:270] Start()
0506 19:56:54.247856 (+ 26us) write_op.cc:276] Timestamp: P: 1715025402885780 usec, L: 0
0506 19:56:54.248010 (+ 154us) write_op.cc:270] Start()
0506 19:56:54.248032 (+ 22us) write_op.cc:276] Timestamp: P: 1715025402886881 usec, L: 0
0506 19:56:54.248182 (+ 150us) write_op.cc:270] Start()
0506 19:56:54.248205 (+ 23us) write_op.cc:276] Timestamp: P: 1715025402887808 usec, L: 0
0506 19:56:54.248343 (+ 138us) write_op.cc:270] Start()
0506 19:56:54.248362 (+ 19us) write_op.cc:276] Timestamp: P: 1715025402894289 usec, L: 0
0506 19:56:54.248494 (+ 132us) write_op.cc:270] Start()
0506 19:56:54.248518 (+ 24us) write_op.cc:276] Timestamp: P: 1715025402898064 usec, L: 0
0506 19:56:54.248659 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.248674 (+ 15us) write_op.cc:276] Timestamp: P: 1715025402898650 usec, L: 0
0506 19:56:54.248783 (+ 109us) write_op.cc:270] Start()
0506 19:56:54.248799 (+ 16us) write_op.cc:276] Timestamp: P: 1715025402899211 usec, L: 0
0506 19:56:54.248898 (+ 99us) write_op.cc:270] Start()
0506 19:56:54.248936 (+ 38us) write_op.cc:276] Timestamp: P: 1715025402923892 usec, L: 0
0506 19:56:54.249135 (+ 199us) write_op.cc:270] Start()
0506 19:56:54.249153 (+ 18us) write_op.cc:276] Timestamp: P: 1715025403013178 usec, L: 0
0506 19:56:54.249288 (+ 135us) write_op.cc:270] Start()
0506 19:56:54.252088 (+ 2800us) write_op.cc:276] Timestamp: P: 1715025403013873 usec, L: 0
0506 19:56:54.252328 (+ 240us) write_op.cc:270] Start()
0506 19:56:54.252358 (+ 30us) write_op.cc:276] Timestamp: P: 1715025403014489 usec, L: 0
0506 19:56:54.252508 (+ 150us) write_op.cc:270] Start()
0506 19:56:54.252534 (+ 26us) write_op.cc:276] Timestamp: P: 1715025403031177 usec, L: 0
0506 19:56:54.252661 (+ 127us) write_op.cc:270] Start()
0506 19:56:54.252684 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403031847 usec, L: 0
0506 19:56:54.252833 (+ 149us) write_op.cc:270] Start()
0506 19:56:54.252853 (+ 20us) write_op.cc:276] Timestamp: P: 1715025403032303 usec, L: 0
0506 19:56:54.252984 (+ 131us) write_op.cc:270] Start()
0506 19:56:54.253012 (+ 28us) write_op.cc:276] Timestamp: P: 1715025403032763 usec, L: 0
0506 19:56:54.253149 (+ 137us) write_op.cc:270] Start()
0506 19:56:54.253171 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403033197 usec, L: 0
0506 19:56:54.253330 (+ 159us) write_op.cc:270] Start()
0506 19:56:54.253354 (+ 24us) write_op.cc:276] Timestamp: P: 1715025403078651 usec, L: 0
0506 19:56:54.253502 (+ 148us) write_op.cc:270] Start()
0506 19:56:54.253525 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403079728 usec, L: 0
0506 19:56:54.253651 (+ 126us) write_op.cc:270] Start()
0506 19:56:54.253673 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403109971 usec, L: 0
0506 19:56:54.253801 (+ 128us) write_op.cc:270] Start()
0506 19:56:54.253823 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403133259 usec, L: 0
0506 19:56:54.253946 (+ 123us) write_op.cc:270] Start()
0506 19:56:54.253998 (+ 52us) write_op.cc:276] Timestamp: P: 1715025403209704 usec, L: 0
0506 19:56:54.254178 (+ 180us) write_op.cc:270] Start()
0506 19:56:54.254203 (+ 25us) write_op.cc:276] Timestamp: P: 1715025403211651 usec, L: 0
0506 19:56:54.254355 (+ 152us) write_op.cc:270] Start()
0506 19:56:54.254378 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403213145 usec, L: 0
0506 19:56:54.254507 (+ 129us) write_op.cc:270] Start()
0506 19:56:54.254529 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403247514 usec, L: 0
0506 19:56:54.254708 (+ 179us) write_op.cc:270] Start()
0506 19:56:54.254730 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403250121 usec, L: 0
0506 19:56:54.254909 (+ 179us) write_op.cc:270] Start()
0506 19:56:54.254932 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403250790 usec, L: 0
0506 19:56:54.255107 (+ 175us) write_op.cc:270] Start()
0506 19:56:54.255130 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403251327 usec, L: 0
0506 19:56:54.255258 (+ 128us) write_op.cc:270] Start()
0506 19:56:54.255280 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403251863 usec, L: 0
0506 19:56:54.255407 (+ 127us) write_op.cc:270] Start()
0506 19:56:54.255430 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403267195 usec, L: 0
0506 19:56:54.255592 (+ 162us) write_op.cc:270] Start()
0506 19:56:54.255615 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403280306 usec, L: 0
0506 19:56:54.255752 (+ 137us) write_op.cc:270] Start()
0506 19:56:54.255774 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403281025 usec, L: 0
0506 19:56:54.255902 (+ 128us) write_op.cc:270] Start()
0506 19:56:54.255925 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403289058 usec, L: 0
0506 19:56:54.256066 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.256088 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403302836 usec, L: 0
0506 19:56:54.256222 (+ 134us) write_op.cc:270] Start()
0506 19:56:54.256244 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403311105 usec, L: 0
0506 19:56:54.256386 (+ 142us) write_op.cc:270] Start()
0506 19:56:54.256408 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403320293 usec, L: 0
0506 19:56:54.256569 (+ 161us) write_op.cc:270] Start()
0506 19:56:54.256593 (+ 24us) write_op.cc:276] Timestamp: P: 1715025403322809 usec, L: 0
0506 19:56:54.256742 (+ 149us) write_op.cc:270] Start()
0506 19:56:54.256765 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403361907 usec, L: 0
0506 19:56:54.256908 (+ 143us) write_op.cc:270] Start()
0506 19:56:54.256930 (+ 22us) write_op.cc:276] Timestamp: P: 1715025403391716 usec, L: 0
0506 19:56:54.265429 (+ 8499us) write_op.cc:270] Start()
0506 19:56:54.265495 (+ 66us) write_op.cc:276] Timestamp: P: 1715025403393194 usec, L: 0
0506 19:56:54.265767 (+ 272us) write_op.cc:270] Start()
0506 19:56:54.265822 (+ 55us) write_op.cc:276] Timestamp: P: 1715025403541472 usec, L: 0
0506 19:56:54.265969 (+ 147us) write_op.cc:270] Start()
0506 19:56:54.265990 (+ 21us) write_op.cc:276] Timestamp: P: 1715025403542213 usec, L: 0
0506 19:56:54.266136 (+ 146us) write_op.cc:270] Start()
0506 19:56:54.266153 (+ 17us) write_op.cc:276] Timestamp: P: 1715025403542806 usec, L: 0
0506 19:56:54.266284 (+ 131us) write_op.cc:270] Start()
0506 19:56:54.266310 (+ 26us) write_op.cc:276] Timestamp: P: 1715025403621949 usec, L: 0
0506 19:56:54.266440 (+ 130us) write_op.cc:270] Start()
0506 19:56:54.266465 (+ 25us) write_op.cc:276] Timestamp: P: 1715025403629271 usec, L: 0
0506 19:56:54.267731 (+ 1266us) write_op.cc:270] Start()
0506 19:56:54.267764 (+ 33us) write_op.cc:276] Timestamp: P: 1715025403630308 usec, L: 0
0506 19:56:54.267996 (+ 232us) write_op.cc:270] Start()
0506 19:56:54.268024 (+ 28us) write_op.cc:276] Timestamp: P: 1715025403631943 usec, L: 0
0506 19:56:54.268178 (+ 154us) write_op.cc:270] Start()
0506 19:56:54.268207 (+ 29us) write_op.cc:276] Timestamp: P: 1715025403652446 usec, L: 0
0506 19:56:54.268304 (+ 97us) write_op.cc:270] Start()
0506 19:56:54.268321 (+ 17us) write_op.cc:276] Timestamp: P: 1715025403684917 usec, L: 0
0506 19:56:54.268419 (+ 98us) write_op.cc:270] Start()
0506 19:56:54.268434 (+ 15us) write_op.cc:276] Timestamp: P: 1715025403754632 usec, L: 0
0506 19:56:54.268532 (+ 98us) write_op.cc:270] Start()
0506 19:56:54.268547 (+ 15us) write_op.cc:276] Timestamp: P: 1715025403756463 usec, L: 0
0506 19:56:54.268683 (+ 136us) write_op.cc:270] Start()
0506 19:56:54.268708 (+ 25us) write_op.cc:276] Timestamp: P: 1715025403760625 usec, L: 0
0506 19:56:54.268821 (+ 113us) write_op.cc:270] Start()
0506 19:56:54.268837 (+ 16us) write_op.cc:276] Timestamp: P: 1715025403764497 usec, L: 0
0506 19:56:54.268978 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.268998 (+ 20us) write_op.cc:276] Timestamp: P: 1715025403765201 usec, L: 0
0506 19:56:54.269129 (+ 131us) write_op.cc:270] Start()
0506 19:56:54.269148 (+ 19us) write_op.cc:276] Timestamp: P: 1715025403765804 usec, L: 0
0506 19:56:54.269266 (+ 118us) write_op.cc:270] Start()
0506 19:56:54.269285 (+ 19us) write_op.cc:276] Timestamp: P: 1715025403777266 usec, L: 0
0506 19:56:54.269425 (+ 140us) write_op.cc:270] Start()
0506 19:56:54.269444 (+ 19us) write_op.cc:276] Timestamp: P: 1715025403777945 usec, L: 0
0506 19:56:54.298566 (+ 29122us) write_op.cc:270] Start()
0506 19:56:54.298624 (+ 58us) write_op.cc:276] Timestamp: P: 1715025403782864 usec, L: 0
0506 19:56:54.299592 (+ 968us) write_op.cc:270] Start()
0506 19:56:54.299624 (+ 32us) write_op.cc:276] Timestamp: P: 1715025403792010 usec, L: 0
0506 19:56:54.299808 (+ 184us) write_op.cc:270] Start()
0506 19:56:54.299831 (+ 23us) write_op.cc:276] Timestamp: P: 1715025403810122 usec, L: 0
0506 19:56:54.300014 (+ 183us) write_op.cc:270] Start()
0506 19:56:54.300042 (+ 28us) write_op.cc:276] Timestamp: P: 1715025403888491 usec, L: 0
0506 19:56:54.300235 (+ 193us) write_op.cc:270] Start()
0506 19:56:54.300267 (+ 32us) write_op.cc:276] Timestamp: P: 1715025403889234 usec, L: 0
0506 19:56:54.300426 (+ 159us) write_op.cc:270] Start()
0506 19:56:54.300444 (+ 18us) write_op.cc:276] Timestamp: P: 1715025403951345 usec, L: 0
0506 19:56:54.300607 (+ 163us) write_op.cc:270] Start()
0506 19:56:54.300626 (+ 19us) write_op.cc:276] Timestamp: P: 1715025403976220 usec, L: 0
0506 19:56:54.300747 (+ 121us) write_op.cc:270] Start()
0506 19:56:54.300765 (+ 18us) write_op.cc:276] Timestamp: P: 1715025403978624 usec, L: 0
0506 19:56:54.300867 (+ 102us) write_op.cc:270] Start()
0506 19:56:54.300885 (+ 18us) write_op.cc:276] Timestamp: P: 1715025403980131 usec, L: 0
0506 19:56:54.301004 (+ 119us) write_op.cc:270] Start()
0506 19:56:54.301025 (+ 21us) write_op.cc:276] Timestamp: P: 1715025404030743 usec, L: 0
0506 19:56:54.301166 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.301186 (+ 20us) write_op.cc:276] Timestamp: P: 1715025404032445 usec, L: 0
0506 19:56:54.301343 (+ 157us) write_op.cc:270] Start()
0506 19:56:54.301376 (+ 33us) write_op.cc:276] Timestamp: P: 1715025404037427 usec, L: 0
0506 19:56:54.301532 (+ 156us) write_op.cc:270] Start()
0506 19:56:54.301556 (+ 24us) write_op.cc:276] Timestamp: P: 1715025404038161 usec, L: 0
0506 19:56:54.301712 (+ 156us) write_op.cc:270] Start()
0506 19:56:54.301738 (+ 26us) write_op.cc:276] Timestamp: P: 1715025404039256 usec, L: 0
0506 19:56:54.301880 (+ 142us) write_op.cc:270] Start()
0506 19:56:54.301902 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404040166 usec, L: 0
0506 19:56:54.302034 (+ 132us) write_op.cc:270] Start()
0506 19:56:54.302075 (+ 41us) write_op.cc:276] Timestamp: P: 1715025404040943 usec, L: 0
0506 19:56:54.302204 (+ 129us) write_op.cc:270] Start()
0506 19:56:54.302222 (+ 18us) write_op.cc:276] Timestamp: P: 1715025404042954 usec, L: 0
0506 19:56:54.302350 (+ 128us) write_op.cc:270] Start()
0506 19:56:54.302372 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404055422 usec, L: 0
0506 19:56:54.302517 (+ 145us) write_op.cc:270] Start()
0506 19:56:54.302540 (+ 23us) write_op.cc:276] Timestamp: P: 1715025404056310 usec, L: 0
0506 19:56:54.306230 (+ 3690us) write_op.cc:270] Start()
0506 19:56:54.306262 (+ 32us) write_op.cc:276] Timestamp: P: 1715025404118433 usec, L: 0
0506 19:56:54.306400 (+ 138us) write_op.cc:270] Start()
0506 19:56:54.306536 (+ 136us) write_op.cc:276] Timestamp: P: 1715025404119091 usec, L: 0
0506 19:56:54.306699 (+ 163us) write_op.cc:270] Start()
0506 19:56:54.306724 (+ 25us) write_op.cc:276] Timestamp: P: 1715025404119696 usec, L: 0
0506 19:56:54.306865 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.306885 (+ 20us) write_op.cc:276] Timestamp: P: 1715025404125171 usec, L: 0
0506 19:56:54.307025 (+ 140us) write_op.cc:270] Start()
0506 19:56:54.307049 (+ 24us) write_op.cc:276] Timestamp: P: 1715025404126060 usec, L: 0
0506 19:56:54.307195 (+ 146us) write_op.cc:270] Start()
0506 19:56:54.307220 (+ 25us) write_op.cc:276] Timestamp: P: 1715025404159932 usec, L: 0
0506 19:56:54.307372 (+ 152us) write_op.cc:270] Start()
0506 19:56:54.307394 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404213582 usec, L: 0
0506 19:56:54.307542 (+ 148us) write_op.cc:270] Start()
0506 19:56:54.307565 (+ 23us) write_op.cc:276] Timestamp: P: 1715025404295522 usec, L: 0
0506 19:56:54.307699 (+ 134us) write_op.cc:270] Start()
0506 19:56:54.307719 (+ 20us) write_op.cc:276] Timestamp: P: 1715025404296439 usec, L: 0
0506 19:56:54.308472 (+ 753us) write_op.cc:270] Start()
0506 19:56:54.308508 (+ 36us) write_op.cc:276] Timestamp: P: 1715025404297731 usec, L: 0
0506 19:56:54.308649 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.308668 (+ 19us) write_op.cc:276] Timestamp: P: 1715025404299958 usec, L: 0
0506 19:56:54.308828 (+ 160us) write_op.cc:270] Start()
0506 19:56:54.308847 (+ 19us) write_op.cc:276] Timestamp: P: 1715025404328279 usec, L: 0
0506 19:56:54.313432 (+ 4585us) write_op.cc:270] Start()
0506 19:56:54.313493 (+ 61us) write_op.cc:276] Timestamp: P: 1715025404338666 usec, L: 0
0506 19:56:54.313820 (+ 327us) write_op.cc:270] Start()
0506 19:56:54.313849 (+ 29us) write_op.cc:276] Timestamp: P: 1715025404339528 usec, L: 0
0506 19:56:54.314078 (+ 229us) write_op.cc:270] Start()
0506 19:56:54.314104 (+ 26us) write_op.cc:276] Timestamp: P: 1715025404340278 usec, L: 0
0506 19:56:54.314242 (+ 138us) write_op.cc:270] Start()
0506 19:56:54.314265 (+ 23us) write_op.cc:276] Timestamp: P: 1715025404350099 usec, L: 0
0506 19:56:54.314409 (+ 144us) write_op.cc:270] Start()
0506 19:56:54.314469 (+ 60us) write_op.cc:276] Timestamp: P: 1715025404370429 usec, L: 0
0506 19:56:54.314613 (+ 144us) write_op.cc:270] Start()
0506 19:56:54.314635 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404392399 usec, L: 0
0506 19:56:54.314758 (+ 123us) write_op.cc:270] Start()
0506 19:56:54.314777 (+ 19us) write_op.cc:276] Timestamp: P: 1715025404404417 usec, L: 0
0506 19:56:54.314906 (+ 129us) write_op.cc:270] Start()
0506 19:56:54.314923 (+ 17us) write_op.cc:276] Timestamp: P: 1715025404405628 usec, L: 0
0506 19:56:54.315043 (+ 120us) write_op.cc:270] Start()
0506 19:56:54.315063 (+ 20us) write_op.cc:276] Timestamp: P: 1715025404408221 usec, L: 0
0506 19:56:54.315196 (+ 133us) write_op.cc:270] Start()
0506 19:56:54.315219 (+ 23us) write_op.cc:276] Timestamp: P: 1715025404410576 usec, L: 0
0506 19:56:54.315359 (+ 140us) write_op.cc:270] Start()
0506 19:56:54.315382 (+ 23us) write_op.cc:276] Timestamp: P: 1715025404463413 usec, L: 0
0506 19:56:54.315516 (+ 134us) write_op.cc:270] Start()
0506 19:56:54.315538 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404572846 usec, L: 0
0506 19:56:54.315708 (+ 170us) write_op.cc:270] Start()
0506 19:56:54.315742 (+ 34us) write_op.cc:276] Timestamp: P: 1715025404573678 usec, L: 0
0506 19:56:54.315882 (+ 140us) write_op.cc:270] Start()
0506 19:56:54.315904 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404661132 usec, L: 0
0506 19:56:54.405979 (+ 90075us) write_op.cc:270] Start()
0506 19:56:54.406026 (+ 47us) write_op.cc:276] Timestamp: P: 1715025404662756 usec, L: 0
0506 19:56:54.406263 (+ 237us) write_op.cc:270] Start()
0506 19:56:54.406290 (+ 27us) write_op.cc:276] Timestamp: P: 1715025404665066 usec, L: 0
0506 19:56:54.406453 (+ 163us) write_op.cc:270] Start()
0506 19:56:54.406478 (+ 25us) write_op.cc:276] Timestamp: P: 1715025404697892 usec, L: 0
0506 19:56:54.406594 (+ 116us) write_op.cc:270] Start()
0506 19:56:54.406612 (+ 18us) write_op.cc:276] Timestamp: P: 1715025404704583 usec, L: 0
0506 19:56:54.406762 (+ 150us) write_op.cc:270] Start()
0506 19:56:54.406788 (+ 26us) write_op.cc:276] Timestamp: P: 1715025404707023 usec, L: 0
0506 19:56:54.406965 (+ 177us) write_op.cc:270] Start()
0506 19:56:54.406989 (+ 24us) write_op.cc:276] Timestamp: P: 1715025404708200 usec, L: 0
0506 19:56:54.407128 (+ 139us) write_op.cc:270] Start()
0506 19:56:54.407152 (+ 24us) write_op.cc:276] Timestamp: P: 1715025404721106 usec, L: 0
0506 19:56:54.407294 (+ 142us) write_op.cc:270] Start()
0506 19:56:54.407316 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404733752 usec, L: 0
0506 19:56:54.407465 (+ 149us) write_op.cc:270] Start()
0506 19:56:54.407487 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404734822 usec, L: 0
0506 19:56:54.407620 (+ 133us) write_op.cc:270] Start()
0506 19:56:54.407642 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404736220 usec, L: 0
0506 19:56:54.407801 (+ 159us) write_op.cc:270] Start()
0506 19:56:54.407823 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404737804 usec, L: 0
0506 19:56:54.407948 (+ 125us) write_op.cc:270] Start()
0506 19:56:54.407970 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404756996 usec, L: 0
0506 19:56:54.408121 (+ 151us) write_op.cc:270] Start()
0506 19:56:54.408145 (+ 24us) write_op.cc:276] Timestamp: P: 1715025404777145 usec, L: 0
0506 19:56:54.408290 (+ 145us) write_op.cc:270] Start()
0506 19:56:54.408312 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404806271 usec, L: 0
0506 19:56:54.408468 (+ 156us) write_op.cc:270] Start()
0506 19:56:54.408490 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404809251 usec, L: 0
0506 19:56:54.408634 (+ 144us) write_op.cc:270] Start()
0506 19:56:54.408656 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404810657 usec, L: 0
0506 19:56:54.408781 (+ 125us) write_op.cc:270] Start()
0506 19:56:54.408803 (+ 22us) write_op.cc:276] Timestamp: P: 1715025404811712 usec, L: 0
0506 19:56:54.409002 (+ 199us) write_op.cc:270] Start()
0506 19:56:54.516144 (+107142us) write_op.cc:276] Timestamp: P: 1715025404824507 usec, L: 0
0506 19:56:54.516440 (+ 296us) write_op.cc:270] Start()
0506 19:56:54.516479 (+ 39us) write_op.cc:276] Timestamp: P: 1715025404849966 usec, L: 0
0506 19:56:54.516648 (+ 169us) write_op.cc:270] Start()
0506 19:56:54.516673 (+ 25us) write_op.cc:276] Timestamp: P: 1715025404868387 usec, L: 0
0506 19:56:54.517058 (+ 385us) write_op.cc:270] Start()
0506 19:56:54.517087 (+ 29us) write_op.cc:276] Timestamp: P: 1715025404883324 usec, L: 0
0506 19:56:54.517247 (+ 160us) write_op.cc:270] Start()
0506 19:56:54.517266 (+ 19us) write_op.cc:276] Timestamp: P: 1715025404891844 usec, L: 0
0506 19:56:54.517421 (+ 155us) write_op.cc:270] Start()
0506 19:56:54.517441 (+ 20us) write_op.cc:276] Timestamp: P: 1715025404896024 usec, L: 0
0506 19:56:54.517549 (+ 108us) write_op.cc:270] Start()
0506 19:56:54.517572 (+ 23us) write_op.cc:276] Timestamp: P: 1715025404920861 usec, L: 0
0506 19:56:54.517677 (+ 105us) write_op.cc:270] Start()
0506 19:56:54.517693 (+ 16us) write_op.cc:276] Timestamp: P: 1715025404922446 usec, L: 0
0506 19:56:54.517790 (+ 97us) write_op.cc:270] Start()
0506 19:56:54.517806 (+ 16us) write_op.cc:276] Timestamp: P: 1715025404943672 usec, L: 0
0506 19:56:54.517934 (+ 128us) write_op.cc:270] Start()
0506 19:56:54.517988 (+ 54us) write_op.cc:276] Timestamp: P: 1715025404944342 usec, L: 0
0506 19:56:54.518125 (+ 137us) write_op.cc:270] Start()
0506 19:56:54.518140 (+ 15us) write_op.cc:276] Timestamp: P: 1715025404944934 usec, L: 0
0506 19:56:54.518253 (+ 113us) write_op.cc:270] Start()
0506 19:56:54.518267 (+ 14us) write_op.cc:276] Timestamp: P: 1715025404951038 usec, L: 0
0506 19:56:54.518367 (+ 100us) write_op.cc:270] Start()
0506 19:56:54.518382 (+ 15us) write_op.cc:276] Timestamp: P: 1715025404951856 usec, L: 0
0506 19:56:54.518495 (+ 113us) write_op.cc:270] Start()
0506 19:56:54.518525 (+ 30us) write_op.cc:276] Timestamp: P: 1715025404952870 usec, L: 0
0506 19:56:54.518631 (+ 106us) write_op.cc:270] Start()
0506 19:56:54.518647 (+ 16us) write_op.cc:276] Timestamp: P: 1715025405013556 usec, L: 0
0506 19:56:54.518741 (+ 94us) write_op.cc:270] Start()
0506 19:56:54.518756 (+ 15us) write_op.cc:276] Timestamp: P: 1715025405014306 usec, L: 0
0506 19:56:54.581844 (+ 63088us) write_op.cc:270] Start()
0506 19:56:54.581899 (+ 55us) write_op.cc:276] Timestamp: P: 1715025405014947 usec, L: 0
0506 19:56:54.582163 (+ 264us) write_op.cc:270] Start()
0506 19:56:54.582188 (+ 25us) write_op.cc:276] Timestamp: P: 1715025405015925 usec, L: 0
0506 19:56:54.582307 (+ 119us) write_op.cc:270] Start()
0506 19:56:54.582327 (+ 20us) write_op.cc:276] Timestamp: P: 1715025405016467 usec, L: 0
0506 19:56:54.582493 (+ 166us) write_op.cc:270] Start()
0506 19:56:54.582515 (+ 22us) write_op.cc:276] Timestamp: P: 1715025405017016 usec, L: 0
0506 19:56:54.582656 (+ 141us) write_op.cc:270] Start()
0506 19:56:54.582675 (+ 19us) write_op.cc:276] Timestamp: P: 1715025405052500 usec, L: 0
0506 19:56:54.582807 (+ 132us) write_op.cc:270] Start()
0506 19:56:54.582834 (+ 27us) write_op.cc:276] Timestamp: P: 1715025405077873 usec, L: 0
0506 19:56:54.582958 (+ 124us) write_op.cc:270] Start()
0506 19:56:54.582976 (+ 18us) write_op.cc:276] Timestamp: P: 1715025405078679 usec, L: 0
0506 19:56:54.583093 (+ 117us) write_op.cc:270] Start()
0506 19:56:54.583111 (+ 18us) write_op.cc:276] Timestamp: P: 1715025405079325 usec, L: 0
0506 19:56:54.583244 (+ 133us) write_op.cc:270] Start()
0506 19:56:54.583263 (+ 19us) write_op.cc:276] Timestamp: P: 1715025405140703 usec, L: 0
0506 19:56:54.583401 (+ 138us) write_op.cc:270] Start()
0506 19:56:54.583423 (+ 22us) write_op.cc:276] Timestamp: P: 1715025405141950 usec, L: 0
0506 19:56:54.583562 (+ 139us) write_op.cc:270] Start()
0506 19:56:54.583581 (+ 19us) write_op.cc:276] Timestamp: P: 1715025405143074 usec, L: 0
0506 19:56:54.583719 (+ 138us) write_op.cc:270] Start()
0506 19:56:54.583737 (+ 18us) write_op.cc:276] Timestamp: P: 1715025405150022 usec, L: 0
0506 19:56:54.583861 (+ 124us) write_op.cc:270] Start()
0506 19:56:54.583880 (+ 19us) write_op.cc:276] Timestamp: P: 1715025405192997 usec, L: 0
0506 19:56:54.583995 (+ 115us) write_op.cc:270] Start()
0506 19:56:54.584018 (+ 23us) write_op.cc:276] Timestamp: P: 1715025405197611 usec, L: 0
0506 19:56:54.584162 (+ 144us) write_op.cc:270] Start()
0506 19:56:54.584180 (+ 18us) write_op.cc:276] Timestamp: P: 1715025405384325 usec, L: 0
0506 19:56:54.651209 (+ 67029us) write_op.cc:270] Start()
0506 19:56:54.651247 (+ 38us) write_op.cc:276] Timestamp: P: 1715025405388594 usec, L: 0
0506 19:56:54.651437 (+ 190us) write_op.cc:270] Start()
0506 19:56:54.651459 (+ 22us) write_op.cc:276] Timestamp: P: 1715025405389226 usec, L: 0
0506 19:56:54.651585 (+ 126us) write_op.cc:270] Start()
0506 19:56:54.651606 (+ 21us) write_op.cc:276] Timestamp: P: 1715025405413217 usec, L: 0
0506 19:56:54.651768 (+ 162us) write_op.cc:270] Start()
0506 19:56:54.651801 (+ 33us) write_op.cc:276] Timestamp: P: 1715025405422239 usec, L: 0
0506 19:56:54.651947 (+ 146us) write_op.cc:270] Start()
0506 19:56:54.651968 (+ 21us) write_op.cc:276] Timestamp: P: 1715025405429364 usec, L: 0
0506 19:56:54.652104 (+ 136us) write_op.cc:270] Start()
0506 19:56:54.652127 (+ 23us) write_op.cc:276] Timestamp: P: 1715025405432830 usec, L: 0
0506 19:56:54.652273 (+ 146us) write_op.cc:270] Start()
0506 19:56:54.652293 (+ 20us) write_op.cc:276] Timestamp: P: 1715025405437750 usec, L: 0
0506 19:56:54.652438 (+ 145us) write_op.cc:270] Start()
0506 19:56:54.652459 (+ 21us) write_op.cc:276] Timestamp: P: 1715025405438424 usec, L: 0
0506 19:56:54.652587 (+ 128us) write_op.cc:270] Start()
0506 19:56:54.652607 (+ 20us) write_op.cc:276] Timestamp: P: 17150
I20240506 19:56:58.858472 13244 consensus_queue.cc:260] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.859104 13244 ts_tablet_manager.cc:1398] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.076s user 0.002s sys 0.000s
I20240506 19:56:58.859329 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 98571fa2168644969c851a8edd343a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 70e43e0a20cf4f05a7bc69278a83d1e3) from {username='slave'} at 127.0.0.1:56326
I20240506 19:56:58.865657 13244 tablet_bootstrap.cc:492] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.867743 13244 tablet_bootstrap.cc:654] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:58.869537 12921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 73289.5us
W20240506 19:56:58.871464 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 56952.9us
I20240506 19:56:58.878216 13245 tablet_bootstrap.cc:492] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.878443 13245 ts_tablet_manager.cc:1367] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.001s
I20240506 19:56:58.879281 13245 raft_consensus.cc:348] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.879559 13245 raft_consensus.cc:374] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.879650 13245 raft_consensus.cc:729] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.879962 12950 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fbbc77c0b64c2fa3d125e074eecd87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.879935 13245 consensus_queue.cc:260] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.880306 12957 tablet_service.cc:1514] Processing DeleteTablet for tablet 20aa06290b9a4814a006c423ccd8fc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-06 19:56:58 UTC) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.880645 13245 ts_tablet_manager.cc:1398] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:58.881008 13245 tablet_bootstrap.cc:492] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:58.881680 12921 consensus_peers.cc:485] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.884471 12949 tablet_service.cc:1514] Processing DeleteTablet for tablet 98571fa2168644969c851a8edd343a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 70e43e0a20cf4f05a7bc69278a83d1e3) from {username='slave'} at 127.0.0.1:41490
I20240506 19:56:58.887105 13245 tablet_bootstrap.cc:654] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.888589 14778 tablet.cc:1722] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240506 19:56:58.894572 14778 tablet.cc:1722] T f3e748d2c32a4b09abc872fed318e643 P 059a5c4a3f7e49ef95af4b8341702745: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:58.895642 14779 tablet.cc:1722] T f3e748d2c32a4b09abc872fed318e643 P 8a9c37db9d4f41cd93820a1c81f48d4d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240506 19:56:58.896117 12922 consensus_peers.cc:485] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1750c4dd8cdc40509c553b63d166c466. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.896299 13245 tablet_bootstrap.cc:492] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.896467 13245 ts_tablet_manager.cc:1367] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240506 19:56:58.896939 14460 consensus_queue.cc:1035] T 7b803ae289344f88b604a2053881d59b P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:58.897217 13245 raft_consensus.cc:348] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.897547 13245 raft_consensus.cc:374] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.897648 13245 raft_consensus.cc:729] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.897898 13245 consensus_queue.cc:260] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:58.898185 12922 consensus_peers.cc:485] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50ef5a2799b447f992207452dbe39645. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.898686 13245 ts_tablet_manager.cc:1398] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:56:58.898996 13245 tablet_bootstrap.cc:492] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:58.907495 14460 consensus_queue.cc:1035] T c659d848fa34442fb605f1757571b5e7 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:58.908315 13245 tablet_bootstrap.cc:654] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:58.915694 13249 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.046s sys 0.024s
W20240506 19:56:58.915885 13249 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.664s user 0.046s sys 0.024s
I20240506 19:56:58.916746 13249 tablet_bootstrap.cc:492] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:58.918005 13249 ts_tablet_manager.cc:1367] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.677s user 0.048s sys 0.025s
I20240506 19:56:58.918332 13246 ts_tablet_manager.cc:1888] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:58.918455 13246 log.cc:1196] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/ddae4b0d8d0943d99f54904266f293d4
I20240506 19:56:58.869275 14766 raft_consensus.cc:482] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.923044 14766 raft_consensus.cc:504] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.923684 13245 tablet_bootstrap.cc:492] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.923838 13245 ts_tablet_manager.cc:1367] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240506 19:56:58.923826 14766 leader_election.cc:288] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:58.898777 12781 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:56:58.924069 12781 leader_election.cc:302] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:58.924688 13245 raft_consensus.cc:348] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.924899 13245 raft_consensus.cc:374] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.924965 13245 raft_consensus.cc:729] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:56:58.925316 12921 consensus_peers.cc:485] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.925168 13245 consensus_queue.cc:260] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.925701 14766 raft_consensus.cc:2726] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:58.925873 13245 ts_tablet_manager.cc:1398] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:58.926136 13245 tablet_bootstrap.cc:492] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:58.928139 12781 leader_election.cc:341] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 17da01620b8b41008421b977455e0c9a
W20240506 19:56:58.928795 12783 leader_election.cc:341] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 17da01620b8b41008421b977455e0c9a
I20240506 19:56:58.929034 12783 leader_election.cc:302] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:58.929406 14766 raft_consensus.cc:2726] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:58.941287 13244 tablet_bootstrap.cc:492] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:58.941517 13244 ts_tablet_manager.cc:1367] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.001s
I20240506 19:56:58.942334 13244 raft_consensus.cc:348] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.942637 13244 raft_consensus.cc:374] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.942749 13244 raft_consensus.cc:729] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.943037 13244 consensus_queue.cc:260] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.943687 13244 ts_tablet_manager.cc:1398] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:58.946259 13244 tablet_bootstrap.cc:492] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:58.948801 13249 raft_consensus.cc:348] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.949033 13249 raft_consensus.cc:374] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.949095 13249 raft_consensus.cc:729] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.949468 13245 tablet_bootstrap.cc:654] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.949395 13249 consensus_queue.cc:260] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.949947 13249 ts_tablet_manager.cc:1398] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240506 19:56:58.959429 13249 tablet_bootstrap.cc:492] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:58.960186 13244 tablet_bootstrap.cc:654] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.960754 13246 ts_tablet_manager.cc:1909] T ddae4b0d8d0943d99f54904266f293d4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:58.961078 13249 tablet_bootstrap.cc:654] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:58.964883 12842 raft_consensus.cc:3037] T 6352d2c630204df3a24e299e167f92d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.965332 13245 tablet_bootstrap.cc:492] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:58.965473 13245 ts_tablet_manager.cc:1367] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.007s
I20240506 19:56:58.966156 13245 raft_consensus.cc:348] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.966337 13245 raft_consensus.cc:374] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:58.966382 13245 raft_consensus.cc:729] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:58.966600 13245 consensus_queue.cc:260] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.967501 13245 ts_tablet_manager.cc:1398] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:58.974865 14785 raft_consensus.cc:482] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:58.975034 14785 raft_consensus.cc:504] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.975737 14785 leader_election.cc:288] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.976114 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b022c086d141a19a9b35c4a8c8bef9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:58.976697 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b022c086d141a19a9b35c4a8c8bef9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:58.976989 12987 raft_consensus.cc:2454] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:58.977366 13057 leader_election.cc:302] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:58.977519 13058 leader_election.cc:341] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.977777 14785 raft_consensus.cc:2781] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:58.977852 14785 raft_consensus.cc:482] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:58.977905 14785 raft_consensus.cc:3037] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.980579 14785 raft_consensus.cc:504] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:58.981666 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b022c086d141a19a9b35c4a8c8bef9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:58.981876 12987 raft_consensus.cc:3037] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:58.981904 14785 leader_election.cc:288] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:58.984440 12987 raft_consensus.cc:2454] T 59b022c086d141a19a9b35c4a8c8bef9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:58.984961 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b022c086d141a19a9b35c4a8c8bef9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:58.985044 13057 leader_election.cc:302] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:58.985317 14785 raft_consensus.cc:2781] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:58.985466 14785 raft_consensus.cc:686] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
W20240506 19:56:58.985597 13058 leader_election.cc:341] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:58.985736 14785 consensus_queue.cc:237] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:58.987804 12921 consensus_peers.cc:485] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.987993 13242 ts_tablet_manager.cc:1888] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:58.988142 13242 log.cc:1196] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/796beb8e0a2c41d698bd4c05796263f1
I20240506 19:56:58.988667 13242 ts_tablet_manager.cc:1909] T 796beb8e0a2c41d698bd4c05796263f1 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:58.984452 13249 tablet_bootstrap.cc:492] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:58.991154 13249 ts_tablet_manager.cc:1367] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240506 19:56:58.998814 14784 consensus_queue.cc:1035] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:59.001147 13249 raft_consensus.cc:348] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.001363 13249 raft_consensus.cc:374] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.001423 13249 raft_consensus.cc:729] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.003013 13249 consensus_queue.cc:260] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.011704 13249 ts_tablet_manager.cc:1398] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240506 19:56:59.016275 13249 tablet_bootstrap.cc:492] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.017810 13249 tablet_bootstrap.cc:654] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.020319 14784 raft_consensus.cc:482] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.020486 14784 raft_consensus.cc:504] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.021248 14784 leader_election.cc:288] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.022028 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:59.022667 12922 leader_election.cc:341] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.023469 13246 tablet_replica.cc:330] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:59.023685 13246 raft_consensus.cc:2229] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.023835 13246 raft_consensus.cc:2258] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.024379 13246 ts_tablet_manager.cc:1875] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.025579 13245 tablet_bootstrap.cc:492] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.027724 13245 tablet_bootstrap.cc:654] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.027766 12921 consensus_peers.cc:485] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.034073 14557 ts_tablet_manager.cc:1888] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:59.039549 13246 ts_tablet_manager.cc:1888] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:59.039695 13246 log.cc:1196] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/4f382a89f94e4fdf9b8266d049635329
I20240506 19:56:59.040030 13246 ts_tablet_manager.cc:1909] T 4f382a89f94e4fdf9b8266d049635329 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:59.041004 13246 tablet_replica.cc:330] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:59.041193 13246 raft_consensus.cc:2229] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.041738 14557 log.cc:1196] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/260d7ab458794df496012cb80466f59c
I20240506 19:56:59.042133 14557 ts_tablet_manager.cc:1909] T 260d7ab458794df496012cb80466f59c P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:59.043315 14557 tablet_replica.cc:330] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:59.043553 14557 raft_consensus.cc:2229] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.043742 14557 raft_consensus.cc:2258] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.044292 14557 ts_tablet_manager.cc:1875] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240506 19:56:59.035543 12922 consensus_peers.cc:485] T 4f382a89f94e4fdf9b8266d049635329 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:59.045881 12921 consensus_peers.cc:485] T 1750c4dd8cdc40509c553b63d166c466 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:58.999740 13244 tablet_bootstrap.cc:492] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.049758 13244 ts_tablet_manager.cc:1367] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.104s user 0.001s sys 0.004s
I20240506 19:56:59.050773 13244 raft_consensus.cc:348] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.051143 13244 raft_consensus.cc:374] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.051292 13244 raft_consensus.cc:729] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.051672 13244 consensus_queue.cc:260] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.028455 14805 tablet.cc:1722] T f3e748d2c32a4b09abc872fed318e643 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:59.045686 14766 raft_consensus.cc:482] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.052532 14766 raft_consensus.cc:504] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.053211 14766 leader_election.cc:288] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.055665 13242 tablet_replica.cc:330] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: stopping tablet replica
I20240506 19:56:59.055871 13242 raft_consensus.cc:2229] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.055809 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b9731a473449b19a9e95653a9d0830" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.055992 12987 raft_consensus.cc:2454] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:56:59.056056 13242 raft_consensus.cc:2258] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.056623 13242 ts_tablet_manager.cc:1875] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.058085 12781 leader_election.cc:302] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:56:59.058554 14766 raft_consensus.cc:2781] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.058701 14766 raft_consensus.cc:482] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.058830 14766 raft_consensus.cc:3037] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.023265 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:59.063915 12921 leader_election.cc:341] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.064049 12921 leader_election.cc:302] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.064482 14784 raft_consensus.cc:2726] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.065057 14766 raft_consensus.cc:504] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.066711 14766 leader_election.cc:288] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.065450 12783 leader_election.cc:341] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 40b9731a473449b19a9e95653a9d0830
I20240506 19:56:59.066751 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b9731a473449b19a9e95653a9d0830" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:59.067518 12987 raft_consensus.cc:3037] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:59.067746 12783 leader_election.cc:341] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 40b9731a473449b19a9e95653a9d0830
I20240506 19:56:59.082290 13246 raft_consensus.cc:2258] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.083169 13246 ts_tablet_manager.cc:1875] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.091534 13249 tablet_bootstrap.cc:492] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.091739 13249 ts_tablet_manager.cc:1367] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.005s
I20240506 19:56:59.092149 13244 ts_tablet_manager.cc:1398] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.042s user 0.002s sys 0.002s
I20240506 19:56:59.092564 13244 tablet_bootstrap.cc:492] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.092586 13245 tablet_bootstrap.cc:492] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.092736 13245 ts_tablet_manager.cc:1367] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.002s
I20240506 19:56:59.093113 13249 raft_consensus.cc:348] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.093334 13249 raft_consensus.cc:374] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.093411 13249 raft_consensus.cc:729] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.093477 13245 raft_consensus.cc:348] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.093664 13245 raft_consensus.cc:374] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.093722 13245 raft_consensus.cc:729] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.094300 13244 tablet_bootstrap.cc:654] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.103363 12987 raft_consensus.cc:2454] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:56:59.103991 12781 leader_election.cc:302] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.105413 14814 raft_consensus.cc:2781] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.105684 14814 raft_consensus.cc:686] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:56:59.105715 13249 consensus_queue.cc:260] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.106010 14814 consensus_queue.cc:237] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.106837 13249 ts_tablet_manager.cc:1398] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240506 19:56:59.107128 13249 tablet_bootstrap.cc:492] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.108471 13249 tablet_bootstrap.cc:654] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.110531 12921 consensus_peers.cc:485] T 50ef5a2799b447f992207452dbe39645 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.124112 13242 ts_tablet_manager.cc:1888] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:59.136974 13249 tablet_bootstrap.cc:492] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.137156 13249 ts_tablet_manager.cc:1367] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.007s
I20240506 19:56:59.138058 13249 raft_consensus.cc:348] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.138271 13249 raft_consensus.cc:374] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.138329 13249 raft_consensus.cc:729] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.138592 13249 consensus_queue.cc:260] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.139329 13249 ts_tablet_manager.cc:1398] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:59.139644 13249 tablet_bootstrap.cc:492] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.140625 13246 ts_tablet_manager.cc:1888] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:59.140781 13246 log.cc:1196] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/bff7050bbc2e47cf823d25ec10b89718
I20240506 19:56:59.141168 13246 ts_tablet_manager.cc:1909] T bff7050bbc2e47cf823d25ec10b89718 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:59.141217 13249 tablet_bootstrap.cc:654] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.142485 13246 tablet_replica.cc:330] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:59.143976 13246 raft_consensus.cc:2229] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.144152 13246 raft_consensus.cc:2258] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.147311 14557 ts_tablet_manager.cc:1888] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:59.147472 14557 log.cc:1196] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/c1ff7a3638004012b08c973192c055a5
I20240506 19:56:59.147848 14557 ts_tablet_manager.cc:1909] T c1ff7a3638004012b08c973192c055a5 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:59.148916 13246 ts_tablet_manager.cc:1875] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.141636 14766 raft_consensus.cc:482] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.157431 14766 raft_consensus.cc:504] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.158212 14766 leader_election.cc:288] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.160060 12781 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 234e4f3439e84b5fbb2f7bab1dbd0822
W20240506 19:56:59.160509 12783 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 234e4f3439e84b5fbb2f7bab1dbd0822
I20240506 19:56:59.160614 12783 leader_election.cc:302] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.161093 14766 raft_consensus.cc:2726] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.145563 13244 tablet_bootstrap.cc:492] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.161547 13244 ts_tablet_manager.cc:1367] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.005s
I20240506 19:56:59.162438 13244 raft_consensus.cc:348] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.162722 13244 raft_consensus.cc:374] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.162994 13244 raft_consensus.cc:729] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.164232 13244 consensus_queue.cc:260] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.164923 13244 ts_tablet_manager.cc:1398] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:56:59.165402 13242 log.cc:1196] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-0/wal/wals/304a1401627948bbaff78fae4d42ab22
I20240506 19:56:59.165874 13242 ts_tablet_manager.cc:1909] T 304a1401627948bbaff78fae4d42ab22 P ef18e88b87064ed4b6c20f9325ebcc33: Deleting consensus metadata
I20240506 19:56:59.165959 13249 tablet_bootstrap.cc:492] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.166123 13249 ts_tablet_manager.cc:1367] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240506 19:56:59.167061 13249 raft_consensus.cc:348] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.167304 13249 raft_consensus.cc:374] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.167358 13249 raft_consensus.cc:729] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.167558 13249 consensus_queue.cc:260] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.168044 13249 ts_tablet_manager.cc:1398] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:59.168323 13249 tablet_bootstrap.cc:492] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.168041 14557 tablet_replica.cc:330] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:59.169186 14557 raft_consensus.cc:2229] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:59.171526 13244 tablet_bootstrap.cc:492] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:59.174875 13058 consensus_peers.cc:485] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.176021 13244 tablet_bootstrap.cc:654] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.173427 12922 consensus_peers.cc:485] T bff7050bbc2e47cf823d25ec10b89718 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bff7050bbc2e47cf823d25ec10b89718. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.179224 14557 raft_consensus.cc:2258] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.180491 14557 ts_tablet_manager.cc:1875] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.181896 14785 raft_consensus.cc:482] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.182039 14785 raft_consensus.cc:504] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.182699 14785 leader_election.cc:288] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.183041 13245 consensus_queue.cc:260] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.184019 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3049a9fa8049e5b84230e5e6e8245c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:59.184562 13058 leader_election.cc:341] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.182400 13249 tablet_bootstrap.cc:654] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.183334 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3049a9fa8049e5b84230e5e6e8245c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.185153 12987 raft_consensus.cc:2454] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:59.185598 13057 leader_election.cc:302] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.187331 14757 raft_consensus.cc:2781] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.187415 14757 raft_consensus.cc:482] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.187474 14757 raft_consensus.cc:3037] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.190037 14757 raft_consensus.cc:504] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.190580 14757 leader_election.cc:288] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.191577 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3049a9fa8049e5b84230e5e6e8245c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:59.191746 12987 raft_consensus.cc:3037] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.194370 12987 raft_consensus.cc:2454] T ff3049a9fa8049e5b84230e5e6e8245c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:59.195423 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3049a9fa8049e5b84230e5e6e8245c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:59.196136 13057 leader_election.cc:302] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:56:59.196442 13058 leader_election.cc:341] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.196748 14785 raft_consensus.cc:2781] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.196861 14785 raft_consensus.cc:686] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:59.197077 14785 consensus_queue.cc:237] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.199295 14766 raft_consensus.cc:482] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.199481 14766 raft_consensus.cc:504] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.200219 14766 leader_election.cc:288] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.201385 12783 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ceeae70e0d1144ea9e3e0f8727c15e6b
W20240506 19:56:59.201758 12781 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:56:59.201874 12781 leader_election.cc:302] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.202350 14766 raft_consensus.cc:2726] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.207193 13246 ts_tablet_manager.cc:1888] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 19:56:59.207314 13246 log.cc:1196] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/880ecc75bab446e39c687f3c8b740361
I20240506 19:56:59.207643 13246 ts_tablet_manager.cc:1909] T 880ecc75bab446e39c687f3c8b740361 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:59.208020 14766 raft_consensus.cc:482] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.208166 14766 raft_consensus.cc:504] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.208765 14766 leader_election.cc:288] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.209820 12783 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 116df9bd6ab04a30ba09861e765d7267
W20240506 19:56:59.210826 12781 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 116df9bd6ab04a30ba09861e765d7267
I20240506 19:56:59.211514 12781 leader_election.cc:302] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.212003 14766 raft_consensus.cc:2726] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.213997 13245 ts_tablet_manager.cc:1398] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.121s user 0.016s sys 0.000s
I20240506 19:56:59.214342 13249 tablet_bootstrap.cc:492] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.214525 13249 ts_tablet_manager.cc:1367] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20240506 19:56:59.215288 13249 raft_consensus.cc:348] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.215466 13249 raft_consensus.cc:374] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.215518 13249 raft_consensus.cc:729] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.215746 13249 consensus_queue.cc:260] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.217778 13249 ts_tablet_manager.cc:1398] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:59.218139 13249 tablet_bootstrap.cc:492] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.221681 14766 raft_consensus.cc:482] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.221860 14766 raft_consensus.cc:504] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.222420 13245 tablet_bootstrap.cc:492] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.222550 14766 leader_election.cc:288] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.223790 13245 tablet_bootstrap.cc:654] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.224078 12783 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 997ecba7fbae41178a25bf7af4d152e1
W20240506 19:56:59.224504 12781 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 997ecba7fbae41178a25bf7af4d152e1
I20240506 19:56:59.224618 12781 leader_election.cc:302] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.225066 14766 raft_consensus.cc:2726] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.226781 13249 tablet_bootstrap.cc:654] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.228263 13246 tablet_replica.cc:330] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: stopping tablet replica
I20240506 19:56:59.228453 13246 raft_consensus.cc:2229] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.228628 13246 raft_consensus.cc:2258] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.230984 13246 ts_tablet_manager.cc:1875] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.231066 13244 tablet_bootstrap.cc:492] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.231235 13244 ts_tablet_manager.cc:1367] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.005s
I20240506 19:56:59.231987 13244 raft_consensus.cc:348] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.232213 13244 raft_consensus.cc:374] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.232295 13244 raft_consensus.cc:729] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.232523 13244 consensus_queue.cc:260] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.233083 13244 ts_tablet_manager.cc:1398] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:59.234956 14557 ts_tablet_manager.cc:1888] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:59.235348 13244 tablet_bootstrap.cc:492] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.237067 13244 tablet_bootstrap.cc:654] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.245389 14557 log.cc:1196] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/304a1401627948bbaff78fae4d42ab22
I20240506 19:56:59.245918 14557 ts_tablet_manager.cc:1909] T 304a1401627948bbaff78fae4d42ab22 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:59.247109 14557 tablet_replica.cc:330] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:59.247371 14557 raft_consensus.cc:2229] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 19:56:59.247558 14557 raft_consensus.cc:2258] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.248142 14557 ts_tablet_manager.cc:1875] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.251048 13249 tablet_bootstrap.cc:492] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.251271 13249 ts_tablet_manager.cc:1367] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240506 19:56:59.252018 13249 raft_consensus.cc:348] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.257167 13245 tablet_bootstrap.cc:492] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.257354 13245 ts_tablet_manager.cc:1367] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20240506 19:56:59.257359 13249 raft_consensus.cc:374] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.257535 13249 raft_consensus.cc:729] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.257908 13249 consensus_queue.cc:260] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.259158 13245 raft_consensus.cc:348] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.259352 13245 raft_consensus.cc:374] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.259402 13245 raft_consensus.cc:729] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.259573 13245 consensus_queue.cc:260] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.257570 13246 ts_tablet_manager.cc:1888] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20240506 19:56:59.258448 13057 consensus_peers.cc:485] T 260d7ab458794df496012cb80466f59c P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 260d7ab458794df496012cb80466f59c. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.259896 13246 log.cc:1196] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-2/wal/wals/20aa06290b9a4814a006c423ccd8fc46
I20240506 19:56:59.260275 13246 ts_tablet_manager.cc:1909] T 20aa06290b9a4814a006c423ccd8fc46 P 8a9c37db9d4f41cd93820a1c81f48d4d: Deleting consensus metadata
I20240506 19:56:59.262037 13245 ts_tablet_manager.cc:1398] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:56:59.263689 13249 ts_tablet_manager.cc:1398] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240506 19:56:59.264039 13249 tablet_bootstrap.cc:492] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.266286 13249 tablet_bootstrap.cc:654] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.277165 13245 tablet_bootstrap.cc:492] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.279104 13244 tablet_bootstrap.cc:492] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.279299 13244 ts_tablet_manager.cc:1367] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.003s
I20240506 19:56:59.280158 13244 raft_consensus.cc:348] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.280390 13244 raft_consensus.cc:374] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.280475 13244 raft_consensus.cc:729] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.280757 13244 consensus_queue.cc:260] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.281816 13245 tablet_bootstrap.cc:654] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.286226 13244 ts_tablet_manager.cc:1398] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240506 19:56:59.286648 13244 tablet_bootstrap.cc:492] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.288880 13244 tablet_bootstrap.cc:654] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.286264 14557 ts_tablet_manager.cc:1888] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 19:56:59.298365 13245 tablet_bootstrap.cc:492] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.298385 13249 tablet_bootstrap.cc:492] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.298522 13245 ts_tablet_manager.cc:1367] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240506 19:56:59.298651 13249 ts_tablet_manager.cc:1367] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20240506 19:56:59.299269 13245 raft_consensus.cc:348] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.299479 13245 raft_consensus.cc:374] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.299328 13249 raft_consensus.cc:348] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.299552 13245 raft_consensus.cc:729] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.299667 13249 raft_consensus.cc:374] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.299753 13249 raft_consensus.cc:729] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.300010 13249 consensus_queue.cc:260] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.303355 13249 ts_tablet_manager.cc:1398] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:56:59.303725 13249 tablet_bootstrap.cc:492] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.297219 14557 log.cc:1196] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/40b9731a473449b19a9e95653a9d0830
I20240506 19:56:59.301144 13245 consensus_queue.cc:260] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.308315 14557 ts_tablet_manager.cc:1909] T 40b9731a473449b19a9e95653a9d0830 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:59.308837 13249 tablet_bootstrap.cc:654] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.312886 14557 tablet_replica.cc:330] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: stopping tablet replica
I20240506 19:56:59.313150 14557 raft_consensus.cc:2229] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Raft consensus shutting down.
I20240506 19:56:59.313553 14557 raft_consensus.cc:2258] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 19:56:59.314473 13245 ts_tablet_manager.cc:1398] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240506 19:56:59.316195 13244 tablet_bootstrap.cc:492] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.316376 13244 ts_tablet_manager.cc:1367] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240506 19:56:59.317548 14557 ts_tablet_manager.cc:1875] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 19:56:59.319519 13244 raft_consensus.cc:348] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.319769 13244 raft_consensus.cc:374] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.319785 13245 tablet_bootstrap.cc:492] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.319857 13244 raft_consensus.cc:729] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.320106 13244 consensus_queue.cc:260] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.321861 13245 tablet_bootstrap.cc:654] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.327255 13249 tablet_bootstrap.cc:492] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.327477 13249 ts_tablet_manager.cc:1367] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240506 19:56:59.328334 13249 raft_consensus.cc:348] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.328599 13249 raft_consensus.cc:374] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.328758 13249 raft_consensus.cc:729] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.329038 13249 consensus_queue.cc:260] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.330050 13244 ts_tablet_manager.cc:1398] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240506 19:56:59.337539 13249 ts_tablet_manager.cc:1398] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240506 19:56:59.339345 13244 tablet_bootstrap.cc:492] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.341270 13249 tablet_bootstrap.cc:492] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.345022 13249 tablet_bootstrap.cc:654] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.346522 13244 tablet_bootstrap.cc:654] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.356463 14557 ts_tablet_manager.cc:1888] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240506 19:56:59.356680 14557 log.cc:1196] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Deleting WAL directory at /tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/ts-1/wal/wals/20aa06290b9a4814a006c423ccd8fc46
I20240506 19:56:59.357132 13244 tablet_bootstrap.cc:492] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.357246 14557 ts_tablet_manager.cc:1909] T 20aa06290b9a4814a006c423ccd8fc46 P 059a5c4a3f7e49ef95af4b8341702745: Deleting consensus metadata
I20240506 19:56:59.357395 13244 ts_tablet_manager.cc:1367] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240506 19:56:59.358176 13244 raft_consensus.cc:348] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.358412 13244 raft_consensus.cc:374] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.358503 13244 raft_consensus.cc:729] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.358814 13244 consensus_queue.cc:260] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.359416 13244 ts_tablet_manager.cc:1398] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:56:59.359776 13244 tablet_bootstrap.cc:492] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.360471 13249 tablet_bootstrap.cc:492] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.360672 13249 ts_tablet_manager.cc:1367] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240506 19:56:59.361497 13249 raft_consensus.cc:348] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.361744 13249 raft_consensus.cc:374] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.361832 13249 raft_consensus.cc:729] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.362066 13249 consensus_queue.cc:260] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.364280 14766 raft_consensus.cc:482] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.364413 14766 raft_consensus.cc:504] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.365041 14766 leader_election.cc:288] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.366510 12783 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d7be9788df8e47d285bd932d078c6a8b
W20240506 19:56:59.368669 12781 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d7be9788df8e47d285bd932d078c6a8b
I20240506 19:56:59.368789 12781 leader_election.cc:302] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.369228 14766 raft_consensus.cc:2726] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.371284 13249 ts_tablet_manager.cc:1398] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240506 19:56:59.376590 13249 tablet_bootstrap.cc:492] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.379081 14784 consensus_queue.cc:1035] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:59.381000 13249 tablet_bootstrap.cc:654] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.387809 14785 raft_consensus.cc:482] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.387972 14785 raft_consensus.cc:504] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.388646 14785 leader_election.cc:288] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.389686 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63baf59dc8244daad831b2987a51cfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.389876 13244 tablet_bootstrap.cc:654] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.390458 13058 leader_election.cc:341] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.390945 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63baf59dc8244daad831b2987a51cfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.391234 12987 raft_consensus.cc:2454] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:59.392199 13249 tablet_bootstrap.cc:492] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.392319 13249 ts_tablet_manager.cc:1367] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240506 19:56:59.393004 13249 raft_consensus.cc:348] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.393195 13249 raft_consensus.cc:374] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.393237 13249 raft_consensus.cc:729] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.393450 13249 consensus_queue.cc:260] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.393939 13249 ts_tablet_manager.cc:1398] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:59.394840 13057 leader_election.cc:302] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.398056 14785 raft_consensus.cc:2781] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.398141 14785 raft_consensus.cc:482] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.398195 14785 raft_consensus.cc:3037] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.399623 14784 raft_consensus.cc:482] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.399737 14784 raft_consensus.cc:504] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.400308 14784 leader_election.cc:288] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.400934 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64485c3c13e94055a019f2ef7be1f2fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:59.401504 12921 leader_election.cc:341] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.401906 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64485c3c13e94055a019f2ef7be1f2fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:59.402443 12922 leader_election.cc:341] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.402551 12922 leader_election.cc:302] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.402635 14785 raft_consensus.cc:504] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.402997 14784 raft_consensus.cc:2726] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.404034 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63baf59dc8244daad831b2987a51cfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:59.404543 13058 leader_election.cc:341] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.404884 13249 tablet_bootstrap.cc:492] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.404964 14785 leader_election.cc:288] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.405225 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63baf59dc8244daad831b2987a51cfb" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:59.405459 12987 raft_consensus.cc:3037] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.405929 13245 tablet_bootstrap.cc:492] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.406922 12845 raft_consensus.cc:3037] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.408058 12987 raft_consensus.cc:2454] T e63baf59dc8244daad831b2987a51cfb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:59.407001 13249 tablet_bootstrap.cc:654] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.410954 12921 consensus_peers.cc:485] T e1b011e94e2b47d7a51d6816576ea124 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.411639 13057 leader_election.cc:302] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.412875 14656 raft_consensus.cc:2781] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.412988 14656 raft_consensus.cc:686] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:59.413218 14656 consensus_queue.cc:237] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.424436 13249 tablet_bootstrap.cc:492] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.424602 13249 ts_tablet_manager.cc:1367] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20240506 19:56:59.425348 13249 raft_consensus.cc:348] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.425562 13249 raft_consensus.cc:374] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.425623 13249 raft_consensus.cc:729] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.425843 13249 consensus_queue.cc:260] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.426457 13249 ts_tablet_manager.cc:1398] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:59.434412 13249 tablet_bootstrap.cc:492] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.435531 13245 ts_tablet_manager.cc:1367] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.116s user 0.018s sys 0.021s
I20240506 19:56:59.436928 13245 raft_consensus.cc:348] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.437431 14805 tablet.cc:1722] T f8ce7f4ceb9d40009e9c75543b1fc545 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:56:59.446143 13249 tablet_bootstrap.cc:654] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.452092 13245 raft_consensus.cc:374] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.452303 13245 raft_consensus.cc:729] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.452637 13245 consensus_queue.cc:260] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.454133 14840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:59.456231 14840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:59.456423 13245 ts_tablet_manager.cc:1398] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240506 19:56:59.456735 13245 tablet_bootstrap.cc:492] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.457598 14833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:59.459810 13245 tablet_bootstrap.cc:654] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.460007 14833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:59.461774 13244 tablet_bootstrap.cc:492] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.461905 13244 ts_tablet_manager.cc:1367] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.102s user 0.027s sys 0.007s
I20240506 19:56:59.462971 13498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97f390a786034cddbff757779037f861" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: false dest_uuid: "27d4482cd59a4e5687992b1f8a177373" is_pre_election: true
I20240506 19:56:59.463287 13498 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97f390a786034cddbff757779037f861 for term 2 because replica has last-logged OpId of term: 1 index: 1445, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 999.
I20240506 19:56:59.463863 14840 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:56:59.466812 13244 raft_consensus.cc:348] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.467005 13244 raft_consensus.cc:374] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.467059 13244 raft_consensus.cc:729] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.468796 13244 consensus_queue.cc:260] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.469358 13244 ts_tablet_manager.cc:1398] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:59.470256 13249 tablet_bootstrap.cc:492] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.470422 13249 ts_tablet_manager.cc:1367] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240506 19:56:59.471136 13249 raft_consensus.cc:348] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.471354 13249 raft_consensus.cc:374] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.471424 13249 raft_consensus.cc:729] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.471685 13249 consensus_queue.cc:260] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.472312 13249 ts_tablet_manager.cc:1398] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:59.472754 13249 tablet_bootstrap.cc:492] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.473516 14833 leader_election.cc:288] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:59.474404 13249 tablet_bootstrap.cc:654] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.480180 14700 raft_consensus.cc:482] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.480371 14700 raft_consensus.cc:504] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.481086 14700 leader_election.cc:288] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.483873 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.484445 13244 tablet_bootstrap.cc:492] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:56:59.484771 12921 leader_election.cc:341] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.485147 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:59.485733 12922 leader_election.cc:341] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.485920 13244 tablet_bootstrap.cc:654] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.485883 12922 leader_election.cc:302] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.486418 14700 raft_consensus.cc:2726] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.488171 14700 raft_consensus.cc:482] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.488313 14700 raft_consensus.cc:504] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.488986 14700 leader_election.cc:288] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.489420 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b625c2bbd89746bc90b8c4c2483b3741" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.489519 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b625c2bbd89746bc90b8c4c2483b3741" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.489629 13116 raft_consensus.cc:2454] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:59.490025 12921 leader_election.cc:341] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.490166 12922 leader_election.cc:302] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.490680 14766 raft_consensus.cc:482] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.490841 14700 raft_consensus.cc:2781] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.490814 14766 raft_consensus.cc:504] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.490953 14700 raft_consensus.cc:482] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.491068 14700 raft_consensus.cc:3037] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.491391 14766 leader_election.cc:288] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.492540 12783 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b2be78cbc8264f6e8d2e05137135161c
W20240506 19:56:59.493415 12781 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b2be78cbc8264f6e8d2e05137135161c
I20240506 19:56:59.493521 12781 leader_election.cc:302] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.493943 14766 raft_consensus.cc:2726] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.494894 14700 raft_consensus.cc:504] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.496430 13244 tablet_bootstrap.cc:492] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.497244 13244 ts_tablet_manager.cc:1367] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240506 19:56:59.497499 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b625c2bbd89746bc90b8c4c2483b3741" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:59.497725 13116 raft_consensus.cc:3037] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.497936 13244 raft_consensus.cc:348] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.498100 13244 raft_consensus.cc:374] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.498142 13244 raft_consensus.cc:729] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.499471 13244 consensus_queue.cc:260] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.500910 13116 raft_consensus.cc:2454] T b625c2bbd89746bc90b8c4c2483b3741 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:59.502015 12922 leader_election.cc:302] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.502430 14739 raft_consensus.cc:2781] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.502586 14739 raft_consensus.cc:686] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:59.502902 14739 consensus_queue.cc:237] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.506068 13244 ts_tablet_manager.cc:1398] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240506 19:56:59.506564 13245 tablet_bootstrap.cc:492] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.506692 13245 ts_tablet_manager.cc:1367] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.007s
I20240506 19:56:59.507385 13245 raft_consensus.cc:348] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.507558 13245 raft_consensus.cc:374] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.507602 13245 raft_consensus.cc:729] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.507812 13245 consensus_queue.cc:260] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.508337 13245 ts_tablet_manager.cc:1398] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:59.508570 13245 tablet_bootstrap.cc:492] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.509505 14700 leader_election.cc:288] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.512616 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b625c2bbd89746bc90b8c4c2483b3741" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:59.513505 12921 leader_election.cc:341] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.517357 13245 tablet_bootstrap.cc:654] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.522934 14656 raft_consensus.cc:482] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.523073 14656 raft_consensus.cc:504] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.523690 14656 leader_election.cc:288] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.525213 13244 tablet_bootstrap.cc:492] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.525733 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d09eafa82a0423a8358958e78f728e2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.525928 12987 raft_consensus.cc:2454] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:59.526660 13057 leader_election.cc:302] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.526906 13244 tablet_bootstrap.cc:654] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.526996 14656 raft_consensus.cc:2781] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.527065 14656 raft_consensus.cc:482] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.527118 14656 raft_consensus.cc:3037] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.529882 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d09eafa82a0423a8358958e78f728e2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:59.530436 13058 leader_election.cc:341] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.531222 14656 raft_consensus.cc:504] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.532289 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d09eafa82a0423a8358958e78f728e2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:59.532466 12987 raft_consensus.cc:3037] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.534394 13245 tablet_bootstrap.cc:492] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.534518 13245 ts_tablet_manager.cc:1367] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20240506 19:56:59.534616 13244 tablet_bootstrap.cc:492] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.534732 13244 ts_tablet_manager.cc:1367] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240506 19:56:59.535087 12987 raft_consensus.cc:2454] T 8d09eafa82a0423a8358958e78f728e2 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:59.535221 13245 raft_consensus.cc:348] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.535396 13245 raft_consensus.cc:374] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.535444 13245 raft_consensus.cc:729] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.535362 13244 raft_consensus.cc:348] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.535589 13244 raft_consensus.cc:374] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.535648 13244 raft_consensus.cc:729] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.535646 13245 consensus_queue.cc:260] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.535794 13057 leader_election.cc:302] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.536149 13245 ts_tablet_manager.cc:1398] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:56:59.536378 14656 leader_election.cc:288] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.536670 14656 raft_consensus.cc:2781] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.537199 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d09eafa82a0423a8358958e78f728e2" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:59.537254 14656 raft_consensus.cc:686] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
W20240506 19:56:59.537837 13058 leader_election.cc:341] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.537571 14656 consensus_queue.cc:237] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.538167 13244 consensus_queue.cc:260] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.541576 14160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 1445 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861" is_pre_election: true
I20240506 19:56:59.541954 14160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d4482cd59a4e5687992b1f8a177373 in term 1.
I20240506 19:56:59.542634 13439 leader_election.cc:302] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d4482cd59a4e5687992b1f8a177373, 97f390a786034cddbff757779037f861; no voters:
I20240506 19:56:59.543474 13245 tablet_bootstrap.cc:492] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.543648 14833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 19:56:59.543767 14833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 190a2eabb30d4154960a44eb50a8a87d)
I20240506 19:56:59.543862 14833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 1 FOLLOWER]: Advancing to term 2
I20240506 19:56:59.548240 14700 raft_consensus.cc:482] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.548385 14700 raft_consensus.cc:504] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.549062 14700 leader_election.cc:288] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.549796 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6245e4adce4038ad6f988877d75045" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.549862 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6245e4adce4038ad6f988877d75045" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.550056 13116 raft_consensus.cc:2454] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:59.550359 12921 leader_election.cc:341] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.550527 12922 leader_election.cc:302] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.550894 14700 raft_consensus.cc:2781] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.550962 14700 raft_consensus.cc:482] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.551020 14700 raft_consensus.cc:3037] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.552662 14739 raft_consensus.cc:482] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.552785 14739 raft_consensus.cc:504] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.553653 14739 leader_election.cc:288] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.553797 14700 raft_consensus.cc:504] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.554088 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21059ce22bff4ccc8ae7d0de0c2fbd7a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.554790 14833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:59.555495 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21059ce22bff4ccc8ae7d0de0c2fbd7a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.555708 13116 raft_consensus.cc:2454] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:56:59.554586 12921 leader_election.cc:341] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.556558 12922 leader_election.cc:302] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.556788 14784 raft_consensus.cc:2781] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.556859 14784 raft_consensus.cc:482] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.556911 14784 raft_consensus.cc:3037] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.557694 13249 tablet_bootstrap.cc:492] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.557711 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6245e4adce4038ad6f988877d75045" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:59.557817 13249 ts_tablet_manager.cc:1367] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.000s
I20240506 19:56:59.557855 13116 raft_consensus.cc:3037] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.558538 13249 raft_consensus.cc:348] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.558734 13249 raft_consensus.cc:374] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.558781 13249 raft_consensus.cc:729] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.558952 13249 consensus_queue.cc:260] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.559455 14784 raft_consensus.cc:504] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.559959 14784 leader_election.cc:288] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.560627 13116 raft_consensus.cc:2454] T cd6245e4adce4038ad6f988877d75045 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:59.561985 13249 ts_tablet_manager.cc:1398] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:56:59.562444 13249 tablet_bootstrap.cc:492] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.562911 14792 raft_consensus.cc:482] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.563023 14792 raft_consensus.cc:504] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.563552 14792 leader_election.cc:288] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.563867 13249 tablet_bootstrap.cc:654] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.564198 14700 leader_election.cc:288] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.564406 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc88443c04b42d19cc7fba339eeeaf5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.565706 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21059ce22bff4ccc8ae7d0de0c2fbd7a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:59.565958 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6245e4adce4038ad6f988877d75045" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:59.566612 12921 leader_election.cc:341] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:59.566792 12921 leader_election.cc:341] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.567095 12922 leader_election.cc:302] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.568046 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc88443c04b42d19cc7fba339eeeaf5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.568235 12986 raft_consensus.cc:2454] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:56:59.568688 14797 raft_consensus.cc:2781] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.568811 14797 raft_consensus.cc:686] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:59.569074 14797 consensus_queue.cc:237] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.569818 13057 leader_election.cc:302] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.570117 14785 raft_consensus.cc:2781] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.570216 14785 raft_consensus.cc:482] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.570287 14785 raft_consensus.cc:3037] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:56:59.570484 13058 leader_election.cc:341] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.572822 14785 raft_consensus.cc:504] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.570065 14792 consensus_queue.cc:1035] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:56:59.574100 14833 leader_election.cc:288] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 election: Requested vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 97f390a786034cddbff757779037f861 (127.12.48.252:36867)
I20240506 19:56:59.574627 13244 ts_tablet_manager.cc:1398] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.040s user 0.001s sys 0.000s
I20240506 19:56:59.576038 14160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27d4482cd59a4e5687992b1f8a177373" candidate_term: 2 candidate_status { last_received { term: 1 index: 1445 } } ignore_live_leader: false dest_uuid: "97f390a786034cddbff757779037f861"
I20240506 19:56:59.576267 14160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 1 FOLLOWER]: Advancing to term 2
I20240506 19:56:59.578788 14785 leader_election.cc:288] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.579350 13244 tablet_bootstrap.cc:492] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.579391 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc88443c04b42d19cc7fba339eeeaf5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:59.579586 12987 raft_consensus.cc:3037] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.580965 13244 tablet_bootstrap.cc:654] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.582267 12987 raft_consensus.cc:2454] T ecc88443c04b42d19cc7fba339eeeaf5 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:59.582659 13057 leader_election.cc:302] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.583742 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc88443c04b42d19cc7fba339eeeaf5" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:59.583920 14785 raft_consensus.cc:2781] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.584048 14785 raft_consensus.cc:686] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:59.584324 14160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d4482cd59a4e5687992b1f8a177373 in term 2.
I20240506 19:56:59.584308 14785 consensus_queue.cc:237] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.585000 13244 tablet_bootstrap.cc:492] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.585114 13244 ts_tablet_manager.cc:1367] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20240506 19:56:59.584316 13058 leader_election.cc:341] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.585788 13244 raft_consensus.cc:348] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.585952 13244 raft_consensus.cc:374] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.585995 13244 raft_consensus.cc:729] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.586154 13244 consensus_queue.cc:260] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.586452 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21059ce22bff4ccc8ae7d0de0c2fbd7a" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:56:59.586611 13116 raft_consensus.cc:3037] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.589264 13116 raft_consensus.cc:2454] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:56:59.589723 12922 leader_election.cc:302] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.590189 14739 raft_consensus.cc:2781] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.590306 14739 raft_consensus.cc:686] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:59.590530 14739 consensus_queue.cc:237] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.591148 13244 ts_tablet_manager.cc:1398] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:56:59.596055 13244 tablet_bootstrap.cc:492] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.597671 13244 tablet_bootstrap.cc:654] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.599014 13439 leader_election.cc:302] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d4482cd59a4e5687992b1f8a177373, 97f390a786034cddbff757779037f861; no voters:
I20240506 19:56:59.599386 14833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 2 FOLLOWER]: Leader election won for term 2
W20240506 19:56:59.608588 13057 consensus_peers.cc:485] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 458314e4653748e48cf646c72373fc3e. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.613864 14833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [term 2 LEADER]: Becoming Leader. State: Replica: 27d4482cd59a4e5687992b1f8a177373, State: Running, Role: LEADER
I20240506 19:56:59.614557 14833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1443, Committed index: 1443, Last appended: 1.1445, Last appended by leader: 1445, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:56:59.616362 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d4482cd59a4e5687992b1f8a177373. Latest consensus state: current_term: 2 leader_uuid: "27d4482cd59a4e5687992b1f8a177373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:56:59.617002 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: LEADER
I20240506 19:56:59.618144 13244 tablet_bootstrap.cc:492] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.618376 13244 ts_tablet_manager.cc:1367] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240506 19:56:59.618808 13249 tablet_bootstrap.cc:492] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.618969 13249 ts_tablet_manager.cc:1367] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.007s
I20240506 19:56:59.619488 14766 raft_consensus.cc:482] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.619590 14766 raft_consensus.cc:504] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.620148 14766 leader_election.cc:288] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.620193 14700 raft_consensus.cc:482] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.620306 14700 raft_consensus.cc:504] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.620901 14700 leader_election.cc:288] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:56:59.621595 12783 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2ff3435c11854649b7daf74f3d8c33d7
W20240506 19:56:59.621855 12781 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2ff3435c11854649b7daf74f3d8c33d7
I20240506 19:56:59.621891 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e6f7555dfa4122ac00bae7b9e733fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.621956 12781 leader_election.cc:302] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.622165 14766 raft_consensus.cc:2726] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.619252 13244 raft_consensus.cc:348] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.622632 13244 raft_consensus.cc:374] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.622696 13244 raft_consensus.cc:729] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:56:59.622892 12921 leader_election.cc:341] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.623155 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e6f7555dfa4122ac00bae7b9e733fe" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.623240 13244 consensus_queue.cc:260] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:56:59.623610 12922 leader_election.cc:341] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.623725 12922 leader_election.cc:302] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.623976 14700 raft_consensus.cc:2726] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:56:59.623683 13058 consensus_peers.cc:485] T f759b560e2ab41a6a076976ccca811d1 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.619652 13249 raft_consensus.cc:348] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.626468 13249 raft_consensus.cc:374] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.626550 13249 raft_consensus.cc:729] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.626778 13249 consensus_queue.cc:260] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.627329 13249 ts_tablet_manager.cc:1398] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240506 19:56:59.627604 13249 tablet_bootstrap.cc:492] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.631711 13249 tablet_bootstrap.cc:654] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.632560 12922 consensus_peers.cc:485] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddae4b0d8d0943d99f54904266f293d4. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:59.636888 13058 consensus_peers.cc:485] T 458314e4653748e48cf646c72373fc3e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.637497 13244 ts_tablet_manager.cc:1398] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240506 19:56:59.641366 13245 tablet_bootstrap.cc:654] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.643667 13244 tablet_bootstrap.cc:492] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.647151 13244 tablet_bootstrap.cc:654] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.655073 13244 tablet_bootstrap.cc:492] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.655073 13249 tablet_bootstrap.cc:492] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.655228 13249 ts_tablet_manager.cc:1367] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.002s
I20240506 19:56:59.655339 13244 ts_tablet_manager.cc:1367] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240506 19:56:59.655885 13249 raft_consensus.cc:348] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.656093 13249 raft_consensus.cc:374] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.655988 13244 raft_consensus.cc:348] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.656167 13244 raft_consensus.cc:374] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.656159 13249 raft_consensus.cc:729] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.656209 13244 raft_consensus.cc:729] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.656370 13249 consensus_queue.cc:260] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.656424 13244 consensus_queue.cc:260] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.658165 13245 tablet_bootstrap.cc:492] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.658315 13245 ts_tablet_manager.cc:1367] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.003s
I20240506 19:56:59.659065 13245 raft_consensus.cc:348] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.659268 13245 raft_consensus.cc:374] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.659330 13245 raft_consensus.cc:729] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.659554 13245 consensus_queue.cc:260] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.664698 13245 ts_tablet_manager.cc:1398] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240506 19:56:59.673177 13245 tablet_bootstrap.cc:492] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.673593 13244 ts_tablet_manager.cc:1398] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240506 19:56:59.673686 13249 ts_tablet_manager.cc:1398] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240506 19:56:59.674018 13249 tablet_bootstrap.cc:492] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.674822 13245 tablet_bootstrap.cc:654] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.675276 13249 tablet_bootstrap.cc:654] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.685876 13244 tablet_bootstrap.cc:492] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.687711 13244 tablet_bootstrap.cc:654] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.688937 13249 tablet_bootstrap.cc:492] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.689461 13249 ts_tablet_manager.cc:1367] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240506 19:56:59.693186 13249 raft_consensus.cc:348] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.693420 13249 raft_consensus.cc:374] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.693473 13249 raft_consensus.cc:729] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.693687 13249 consensus_queue.cc:260] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.694309 13249 ts_tablet_manager.cc:1398] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:56:59.694391 13245 tablet_bootstrap.cc:492] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.694559 13245 ts_tablet_manager.cc:1367] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240506 19:56:59.695485 13245 raft_consensus.cc:348] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.695746 13245 raft_consensus.cc:374] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.695822 13245 raft_consensus.cc:729] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.696147 13245 consensus_queue.cc:260] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.696714 14797 raft_consensus.cc:482] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.696851 14797 raft_consensus.cc:504] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.697432 14797 leader_election.cc:288] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.700940 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:59.701738 12921 leader_election.cc:341] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.702644 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:56:59.703130 12922 leader_election.cc:341] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.703222 12922 leader_election.cc:302] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.703603 14797 raft_consensus.cc:2726] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.704195 13249 tablet_bootstrap.cc:492] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.705816 13245 ts_tablet_manager.cc:1398] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240506 19:56:59.709448 13249 tablet_bootstrap.cc:654] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.711750 13245 tablet_bootstrap.cc:492] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.722312 13244 tablet_bootstrap.cc:492] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.722514 13244 ts_tablet_manager.cc:1367] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.005s
I20240506 19:56:59.723037 13245 tablet_bootstrap.cc:654] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.729611 13244 raft_consensus.cc:348] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.730152 13244 raft_consensus.cc:374] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.730216 13244 raft_consensus.cc:729] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.730432 13244 consensus_queue.cc:260] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.735263 14797 raft_consensus.cc:482] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.735445 14797 raft_consensus.cc:504] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.736129 14797 leader_election.cc:288] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.739158 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.739588 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:56:59.740267 12921 leader_election.cc:341] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:56:59.740710 12922 leader_election.cc:341] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.740803 12922 leader_election.cc:302] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.741565 14797 raft_consensus.cc:2726] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.745206 13244 ts_tablet_manager.cc:1398] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20240506 19:56:59.746425 13249 tablet_bootstrap.cc:492] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.746603 13249 ts_tablet_manager.cc:1367] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.019s
I20240506 19:56:59.747268 13249 raft_consensus.cc:348] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.747452 13249 raft_consensus.cc:374] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.747574 13249 raft_consensus.cc:729] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.747874 13249 consensus_queue.cc:260] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.753746 13249 ts_tablet_manager.cc:1398] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:56:59.754050 13249 tablet_bootstrap.cc:492] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.755601 13249 tablet_bootstrap.cc:654] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.763775 13244 tablet_bootstrap.cc:492] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.765372 14766 raft_consensus.cc:482] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.765599 13244 tablet_bootstrap.cc:654] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.765523 14766 raft_consensus.cc:504] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.766201 14766 leader_election.cc:288] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.767298 12783 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: da06187c83b94d03ae4ebb67a2873cf0
W20240506 19:56:59.767437 12781 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:56:59.767529 12781 leader_election.cc:302] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.767982 14766 raft_consensus.cc:2726] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.774380 13249 tablet_bootstrap.cc:492] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.774582 13249 ts_tablet_manager.cc:1367] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20240506 19:56:59.775557 13249 raft_consensus.cc:348] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.775844 13249 raft_consensus.cc:374] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.775921 13249 raft_consensus.cc:729] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.776212 13249 consensus_queue.cc:260] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.785216 13244 tablet_bootstrap.cc:492] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.785671 13249 ts_tablet_manager.cc:1398] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240506 19:56:59.785758 13245 tablet_bootstrap.cc:492] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.785902 13245 ts_tablet_manager.cc:1367] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.002s
I20240506 19:56:59.786029 13249 tablet_bootstrap.cc:492] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.786674 13245 raft_consensus.cc:348] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.786875 13245 raft_consensus.cc:374] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.786923 13245 raft_consensus.cc:729] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.787135 13245 consensus_queue.cc:260] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.787639 13249 tablet_bootstrap.cc:654] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.787663 13245 ts_tablet_manager.cc:1398] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:56:59.787948 13245 tablet_bootstrap.cc:492] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.789284 13245 tablet_bootstrap.cc:654] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.799718 13244 ts_tablet_manager.cc:1367] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.006s
I20240506 19:56:59.801404 13244 raft_consensus.cc:348] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.801613 13244 raft_consensus.cc:374] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.801671 13244 raft_consensus.cc:729] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.802520 13244 consensus_queue.cc:260] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.803040 13244 ts_tablet_manager.cc:1398] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 19:56:59.803308 13244 tablet_bootstrap.cc:492] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.804833 13244 tablet_bootstrap.cc:654] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.808163 12845 raft_consensus.cc:3037] T 6f4e97d10ae54e558920d116d8fa0ba5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.817605 13249 tablet_bootstrap.cc:492] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.818547 13249 ts_tablet_manager.cc:1367] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.004s
I20240506 19:56:59.819741 13249 raft_consensus.cc:348] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.821501 13249 raft_consensus.cc:374] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.821810 13249 raft_consensus.cc:729] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.822235 13249 consensus_queue.cc:260] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.822568 13245 tablet_bootstrap.cc:492] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.822757 13245 ts_tablet_manager.cc:1367] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.005s
I20240506 19:56:59.823597 13245 raft_consensus.cc:348] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.823771 13245 raft_consensus.cc:374] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.823817 13245 raft_consensus.cc:729] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.824040 13245 consensus_queue.cc:260] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.830991 14739 raft_consensus.cc:482] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.831130 14739 raft_consensus.cc:504] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.831729 14739 leader_election.cc:288] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.833498 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20016b56be94248905e0e8d6a35100f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.833699 13116 raft_consensus.cc:2454] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:56:59.834128 12922 leader_election.cc:302] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.834422 14797 raft_consensus.cc:2781] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.834493 14797 raft_consensus.cc:482] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.834553 14797 raft_consensus.cc:3037] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.834972 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20016b56be94248905e0e8d6a35100f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.837136 14797 raft_consensus.cc:504] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.842855 14797 leader_election.cc:288] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:56:59.843421 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20016b56be94248905e0e8d6a35100f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:56:59.843484 12483 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu pid 12491: Timed out: CheckLeaks RPC to 127.12.48.254:33495 timed out after 1.000s (SENT)
I20240506 19:56:59.843605 13116 raft_consensus.cc:3037] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.843640 12483 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu with pid 12491
I20240506 19:56:59.846385 13116 raft_consensus.cc:2454] T a20016b56be94248905e0e8d6a35100f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:56:59.837651 12921 leader_election.cc:341] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.847452 12922 leader_election.cc:302] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:56:59.848564 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20016b56be94248905e0e8d6a35100f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:56:59.849114 12921 leader_election.cc:341] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.849335 14700 raft_consensus.cc:2781] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.849455 14700 raft_consensus.cc:686] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:56:59.849689 14700 consensus_queue.cc:237] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.855180 13245 ts_tablet_manager.cc:1398] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20240506 19:56:59.856328 13249 ts_tablet_manager.cc:1398] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.037s user 0.002s sys 0.001s
I20240506 19:56:59.856626 13249 tablet_bootstrap.cc:492] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.857784 13245 tablet_bootstrap.cc:492] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.863421 13245 tablet_bootstrap.cc:654] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.864907 13244 tablet_bootstrap.cc:492] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.865059 13249 tablet_bootstrap.cc:654] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.865075 13244 ts_tablet_manager.cc:1367] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.006s
I20240506 19:56:59.865801 13244 raft_consensus.cc:348] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.865988 13244 raft_consensus.cc:374] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.866037 13244 raft_consensus.cc:729] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.866292 13244 consensus_queue.cc:260] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.873526 13244 ts_tablet_manager.cc:1398] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240506 19:56:59.878057 13244 tablet_bootstrap.cc:492] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.879891 13244 tablet_bootstrap.cc:654] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.885555 13244 tablet_bootstrap.cc:492] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.885706 13244 ts_tablet_manager.cc:1367] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240506 19:56:59.886437 13244 raft_consensus.cc:348] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.886634 13244 raft_consensus.cc:374] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.886682 13244 raft_consensus.cc:729] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.886890 13244 consensus_queue.cc:260] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.887857 13249 tablet_bootstrap.cc:492] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.887993 13249 ts_tablet_manager.cc:1367] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
I20240506 19:56:59.888638 13249 raft_consensus.cc:348] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.888842 13249 raft_consensus.cc:374] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.888898 13249 raft_consensus.cc:729] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.889123 13249 consensus_queue.cc:260] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.889652 13249 ts_tablet_manager.cc:1398] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:56:59.891786 13245 tablet_bootstrap.cc:492] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.891930 13245 ts_tablet_manager.cc:1367] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.003s
I20240506 19:56:59.892670 13245 raft_consensus.cc:348] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.892848 13245 raft_consensus.cc:374] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.892897 13245 raft_consensus.cc:729] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.893177 13245 consensus_queue.cc:260] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.893826 13245 ts_tablet_manager.cc:1398] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:56:59.897039 13249 tablet_bootstrap.cc:492] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:56:59.898753 13249 tablet_bootstrap.cc:654] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.903044 13244 ts_tablet_manager.cc:1398] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
W20240506 19:56:59.914744 12921 consensus_peers.cc:485] T ddae4b0d8d0943d99f54904266f293d4 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.915382 13244 tablet_bootstrap.cc:492] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.917055 13244 tablet_bootstrap.cc:654] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.918545 13245 tablet_bootstrap.cc:492] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:56:59.920684 13245 tablet_bootstrap.cc:654] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.925232 13244 tablet_bootstrap.cc:492] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:56:59.925396 13244 ts_tablet_manager.cc:1367] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240506 19:56:59.926127 13244 raft_consensus.cc:348] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.926398 13244 raft_consensus.cc:374] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.926455 13244 raft_consensus.cc:729] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.926810 13244 consensus_queue.cc:260] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.930441 14766 raft_consensus.cc:482] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.930569 14766 raft_consensus.cc:504] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.931159 14766 leader_election.cc:288] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:56:59.932129 12781 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 452a83470585433e8def7174300d3a7f
W20240506 19:56:59.933368 12783 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 452a83470585433e8def7174300d3a7f
I20240506 19:56:59.933482 12783 leader_election.cc:302] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.933925 14766 raft_consensus.cc:2726] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.937121 13244 ts_tablet_manager.cc:1398] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240506 19:56:59.938578 13245 tablet_bootstrap.cc:492] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:56:59.938759 13245 ts_tablet_manager.cc:1367] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240506 19:56:59.942519 13244 tablet_bootstrap.cc:492] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:56:59.939484 13245 raft_consensus.cc:348] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.944416 14656 raft_consensus.cc:482] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.944571 12845 raft_consensus.cc:3037] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.944591 14656 raft_consensus.cc:504] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.945197 14656 leader_election.cc:288] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.945742 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:56:59.946102 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.946328 12986 raft_consensus.cc:2454] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:56:59.946655 13058 leader_election.cc:341] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.946986 13057 leader_election.cc:302] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.947399 14656 raft_consensus.cc:2781] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:56:59.947468 14656 raft_consensus.cc:482] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:56:59.947526 14656 raft_consensus.cc:3037] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.950045 14656 raft_consensus.cc:504] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.950858 14656 leader_election.cc:288] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.952121 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:56:59.953065 13244 tablet_bootstrap.cc:654] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:56:59.953441 12922 consensus_peers.cc:485] T 880ecc75bab446e39c687f3c8b740361 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 880ecc75bab446e39c687f3c8b740361. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:56:59.953524 13058 leader_election.cc:341] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:56:59.953441 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:56:59.953919 12986 raft_consensus.cc:3037] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:56:59.957274 12986 raft_consensus.cc:2454] T 85365758f6a64536ab61e8349de63dea P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:56:59.957929 13057 leader_election.cc:302] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:56:59.958544 14656 raft_consensus.cc:2781] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:56:59.959735 14656 raft_consensus.cc:686] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:56:59.962507 14656 consensus_queue.cc:237] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.967368 13249 tablet_bootstrap.cc:492] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:56:59.967576 13249 ts_tablet_manager.cc:1367] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.000s
I20240506 19:56:59.968410 13249 raft_consensus.cc:348] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.969050 13249 raft_consensus.cc:374] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.969259 13249 raft_consensus.cc:729] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.969708 13249 consensus_queue.cc:260] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.973201 14884 tablet.cc:1722] T 4973e600038742a6bec0bb2d8f3551fe P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240506 19:56:59.978648 13249 ts_tablet_manager.cc:1398] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240506 19:56:59.979300 13245 raft_consensus.cc:374] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:56:59.979399 13245 raft_consensus.cc:729] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:56:59.989136 13249 tablet_bootstrap.cc:492] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:56:59.992411 13058 consensus_peers.cc:485] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:56:59.992564 13245 consensus_queue.cc:260] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:56:59.981485 14766 raft_consensus.cc:482] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:56:59.993352 14766 raft_consensus.cc:504] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:56:59.994006 14766 leader_election.cc:288] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:56:59.993286 13245 ts_tablet_manager.cc:1398] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.054s user 0.001s sys 0.001s
I20240506 19:56:59.994661 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:56:59.995334 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ca9a1eaffc45f486d7a7e937a3e787" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:56:59.996196 12783 leader_election.cc:302] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:56:59.996627 14766 raft_consensus.cc:2726] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:56:59.997524 13249 tablet_bootstrap.cc:654] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:56:59.998479 13245 tablet_bootstrap.cc:492] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:56:59.999276 13442 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.48.254:33495 timed out after 0.526s (SENT) [suppressed 1 similar messages]
I20240506 19:57:00.001619 14766 raft_consensus.cc:482] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.001756 14766 raft_consensus.cc:504] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.002311 14766 leader_election.cc:288] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.003281 12783 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 60d1f01c0edb4f6b8a621c1643ca1afb
W20240506 19:57:00.004662 12781 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:57:00.004776 12781 leader_election.cc:302] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.005229 14766 raft_consensus.cc:2726] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.006911 13245 tablet_bootstrap.cc:654] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.008671 13244 tablet_bootstrap.cc:492] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.008792 13244 ts_tablet_manager.cc:1367] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.005s
I20240506 19:57:00.009513 13244 raft_consensus.cc:348] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.015457 13249 tablet_bootstrap.cc:492] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.015607 13249 ts_tablet_manager.cc:1367] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240506 19:57:00.016438 13244 raft_consensus.cc:374] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.016521 13244 raft_consensus.cc:729] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.016356 13249 raft_consensus.cc:348] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.016865 13249 raft_consensus.cc:374] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.016916 13249 raft_consensus.cc:729] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.017159 13244 consensus_queue.cc:260] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.017519 13249 consensus_queue.cc:260] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.020931 13244 ts_tablet_manager.cc:1398] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240506 19:57:00.021243 13244 tablet_bootstrap.cc:492] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:00.022367 12923 connection.cc:535] server connection from 127.0.0.1:41490 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.022855 12783 connection.cc:535] server connection from 127.0.0.1:50504 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 19:57:00.023006 13244 tablet_bootstrap.cc:654] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:00.023355 13059 connection.cc:535] server connection from 127.0.0.1:56326 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.024250 13182 connection.cc:535] client connection to 127.12.48.254:33495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.025605 13442 connection.cc:535] client connection to 127.12.48.254:33495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.025796 13182 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.48.254:33495)
W20240506 19:57:00.025841 13442 leader_election.cc:334] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.026562 13182 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.48.254:33495)
W20240506 19:57:00.027180 13182 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.48.254:33495)
W20240506 19:57:00.028092 13182 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.48.254:33495)
I20240506 19:57:00.029711 14700 raft_consensus.cc:482] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.029847 14700 raft_consensus.cc:504] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.030423 14700 leader_election.cc:288] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:00.030910 14766 raft_consensus.cc:482] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.031000 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ece0bd3a3b47ef88a0b4910f0efae9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.031046 14766 raft_consensus.cc:504] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.031219 13116 raft_consensus.cc:2454] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:00.031620 14766 leader_election.cc:288] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:00.031677 12922 leader_election.cc:302] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.031991 14700 raft_consensus.cc:2781] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.032064 14700 raft_consensus.cc:482] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.032148 14700 raft_consensus.cc:3037] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.034792 14700 raft_consensus.cc:504] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.035912 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ece0bd3a3b47ef88a0b4910f0efae9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.036523 12921 leader_election.cc:341] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:00.036824 12781 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a1160ab972124630a9be7abeefb4aa56
I20240506 19:57:00.037604 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ece0bd3a3b47ef88a0b4910f0efae9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.037918 14700 leader_election.cc:288] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:57:00.038354 12921 leader_election.cc:341] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.039023 13249 ts_tablet_manager.cc:1398] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240506 19:57:00.041921 13249 tablet_bootstrap.cc:492] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:00.042428 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ece0bd3a3b47ef88a0b4910f0efae9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:00.042640 13116 raft_consensus.cc:3037] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.043604 13249 tablet_bootstrap.cc:654] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.045161 13245 tablet_bootstrap.cc:492] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:57:00.045703 12783 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a1160ab972124630a9be7abeefb4aa56
I20240506 19:57:00.045840 12783 leader_election.cc:302] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.046089 14766 raft_consensus.cc:2726] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.045359 13245 ts_tablet_manager.cc:1367] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240506 19:57:00.046916 13116 raft_consensus.cc:2454] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:00.047286 12922 leader_election.cc:302] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.047554 14797 raft_consensus.cc:2781] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.047814 14797 raft_consensus.cc:686] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:00.048091 14797 consensus_queue.cc:237] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
W20240506 19:57:00.050552 13060 connection.cc:535] client connection to 127.12.48.254:33495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.052569 12923 connection.cc:535] client connection to 127.12.48.254:33495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.053905 12784 connection.cc:535] client connection to 127.12.48.254:33495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.054939 14089 connection.cc:535] client connection to 127.12.48.254:33495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.055222 14089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 19:57:00.056193 14766 raft_consensus.cc:482] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.056331 14766 raft_consensus.cc:504] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.056950 14766 leader_election.cc:288] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.057528 13439 connection.cc:535] server connection from 127.0.0.1:36542 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.057966 14844 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 19:57:00.027094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.48.254:33495 (local address 127.0.0.1:47928)
0506 19:57:00.027279 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0506 19:57:00.027301 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0506 19:57:00.027418 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 19:57:00.057633 (+ 30215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20240506 19:57:00.058180 13442 leader_election.cc:334] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 19:57:00.058706 12781 leader_election.cc:341] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 673dc7c83f8748a09c41d51b42214aaf
I20240506 19:57:00.061851 14766 raft_consensus.cc:482] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.061996 14766 raft_consensus.cc:504] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.062650 14766 leader_election.cc:288] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.065353 13245 raft_consensus.cc:348] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.065564 13245 raft_consensus.cc:374] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.065613 13245 raft_consensus.cc:729] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
W20240506 19:57:00.066594 12781 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:57:00.066695 13245 consensus_queue.cc:260] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.067270 13245 ts_tablet_manager.cc:1398] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240506 19:57:00.069013 14656 raft_consensus.cc:482] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.069160 14656 raft_consensus.cc:504] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.069825 14656 leader_election.cc:288] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.070573 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.070784 12986 raft_consensus.cc:2454] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:00.071195 13057 leader_election.cc:302] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.071868 13245 tablet_bootstrap.cc:492] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:00.073009 14893 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 19:57:00.031473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.48.254:33495 (local address 127.0.0.1:47936)
0506 19:57:00.046783 (+ 15310us) negotiation.cc:107] Waiting for socket to connect
0506 19:57:00.046804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0506 19:57:00.046942 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 19:57:00.070989 (+ 24047us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15107,"thread_start_us":27479,"threads_started":1}
I20240506 19:57:00.073491 13245 tablet_bootstrap.cc:654] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.076086 13244 tablet_bootstrap.cc:492] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.076222 13244 ts_tablet_manager.cc:1367] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20240506 19:57:00.076867 14656 raft_consensus.cc:2781] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.076949 14656 raft_consensus.cc:482] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.077013 14656 raft_consensus.cc:3037] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.077015 13244 raft_consensus.cc:348] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.077199 13244 raft_consensus.cc:374] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.077256 13244 raft_consensus.cc:729] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.079674 14656 raft_consensus.cc:504] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.080407 14739 raft_consensus.cc:482] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.080552 14739 raft_consensus.cc:504] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.081125 14739 leader_election.cc:288] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.081830 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85233b180f4b30a86bb65c86af684e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.082535 12921 leader_election.cc:341] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.082572 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.083097 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.083587 13058 leader_election.cc:341] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:00.083904 13058 leader_election.cc:341] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.086350 13249 tablet_bootstrap.cc:492] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.086494 13249 ts_tablet_manager.cc:1367] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240506 19:57:00.087283 13249 raft_consensus.cc:348] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.090371 13249 raft_consensus.cc:374] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.090457 13249 raft_consensus.cc:729] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.090780 13244 consensus_queue.cc:260] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.091457 13249 consensus_queue.cc:260] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.092037 13249 ts_tablet_manager.cc:1398] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:00.092352 13249 tablet_bootstrap.cc:492] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:00.097892 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85233b180f4b30a86bb65c86af684e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.098171 13116 raft_consensus.cc:2454] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:00.099098 12922 leader_election.cc:302] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.099483 14797 raft_consensus.cc:2781] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.099545 14797 raft_consensus.cc:482] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.099647 14797 raft_consensus.cc:3037] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.101119 13244 ts_tablet_manager.cc:1398] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
W20240506 19:57:00.103087 13442 consensus_peers.cc:485] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 -> Peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Couldn't send request to peer 190a2eabb30d4154960a44eb50a8a87d. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.103390 14797 raft_consensus.cc:504] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.103637 14766 raft_consensus.cc:482] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.103787 14766 raft_consensus.cc:504] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.104355 14766 leader_election.cc:288] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.104560 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85233b180f4b30a86bb65c86af684e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:00.104773 13116 raft_consensus.cc:3037] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.105031 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85233b180f4b30a86bb65c86af684e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.105703 12921 leader_election.cc:341] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.106135 14797 leader_election.cc:288] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:00.106587 12781 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 08dafb661b2443f5afadc6816868bac8
I20240506 19:57:00.107440 14160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Refusing update from remote peer 27d4482cd59a4e5687992b1f8a177373: Log matching property violated. Preceding OpId in replica: term: 1 index: 999. Preceding OpId from leader: term: 2 index: 1446. (index mismatch)
I20240506 19:57:00.108119 14858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1446, Last known committed idx: 995, Time since last communication: 0.000s
I20240506 19:57:00.117041 14797 raft_consensus.cc:482] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.118101 14797 raft_consensus.cc:504] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.119432 13244 tablet_bootstrap.cc:492] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:00.119889 14089 leader_election.cc:334] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111)
I20240506 19:57:00.120133 14089 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:57:00.119606 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.123965 12921 leader_election.cc:341] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.126645 14797 leader_election.cc:288] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:00.130765 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:00.132102 12922 leader_election.cc:341] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.132071 14785 consensus_queue.cc:1035] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.132201 12922 leader_election.cc:302] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.132642 14739 raft_consensus.cc:2726] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:00.133386 12783 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:57:00.133515 12783 leader_election.cc:302] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.134069 14766 raft_consensus.cc:2726] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.135468 14656 leader_election.cc:288] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.135617 12783 leader_election.cc:341] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 673dc7c83f8748a09c41d51b42214aaf
I20240506 19:57:00.135761 12783 leader_election.cc:302] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.136287 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.136507 12987 raft_consensus.cc:3037] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.137933 13116 raft_consensus.cc:2454] T bc85233b180f4b30a86bb65c86af684e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:00.138479 12922 leader_election.cc:302] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.138921 14797 raft_consensus.cc:2781] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.139175 14797 raft_consensus.cc:686] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:00.139470 14797 consensus_queue.cc:237] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.139721 12987 raft_consensus.cc:2454] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.139962 14766 raft_consensus.cc:2726] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.140458 13057 leader_election.cc:302] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:57:00.140830 12783 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 08dafb661b2443f5afadc6816868bac8
I20240506 19:57:00.140964 12783 leader_election.cc:302] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.141415 14766 raft_consensus.cc:2726] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.141690 14766 raft_consensus.cc:482] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.141785 14766 raft_consensus.cc:504] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.142417 14766 leader_election.cc:288] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.143312 12781 leader_election.cc:341] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 04738def56324a368f8182543107e32e
W20240506 19:57:00.143818 14896 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 19:57:00.052114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.48.254:33495 (local address 127.12.48.195:34825)
0506 19:57:00.134864 (+ 82750us) negotiation.cc:107] Waiting for socket to connect
0506 19:57:00.134979 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":82641,"thread_start_us":78077,"threads_started":1}
I20240506 19:57:00.143860 14785 raft_consensus.cc:2781] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.143826 14656 raft_consensus.cc:482] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.144244 14656 raft_consensus.cc:504] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.144845 14656 leader_election.cc:288] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.144874 12783 leader_election.cc:341] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 04738def56324a368f8182543107e32e
I20240506 19:57:00.144955 12783 leader_election.cc:302] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.145316 14766 raft_consensus.cc:2726] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.145720 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.146263 13058 leader_election.cc:341] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.146551 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.146795 12987 raft_consensus.cc:2454] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:00.147133 13057 leader_election.cc:302] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.147532 14656 raft_consensus.cc:2781] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.147611 14656 raft_consensus.cc:482] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.147676 14656 raft_consensus.cc:3037] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.150195 14656 raft_consensus.cc:504] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.150813 13249 tablet_bootstrap.cc:654] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.150862 12842 raft_consensus.cc:3037] T 10ca9a1eaffc45f486d7a7e937a3e787 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.152446 13244 tablet_bootstrap.cc:654] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.155365 14656 leader_election.cc:288] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.155661 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:00.156093 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.156273 12987 raft_consensus.cc:3037] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:00.156582 13058 leader_election.cc:341] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.168772 14766 raft_consensus.cc:482] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.168951 14766 raft_consensus.cc:504] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.169605 14766 leader_election.cc:288] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:57:00.176719 12783 consensus_peers.cc:485] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40b9731a473449b19a9e95653a9d0830. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:00.176966 12783 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:57:00.178292 14785 raft_consensus.cc:686] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.178682 14785 consensus_queue.cc:237] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.186021 12987 raft_consensus.cc:2454] T 5fe8d4891c484627ac729971ff7aad3e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.186671 13057 leader_election.cc:302] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.187250 14904 raft_consensus.cc:2781] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.187546 14904 raft_consensus.cc:686] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.187894 14904 consensus_queue.cc:237] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.191375 13245 tablet_bootstrap.cc:492] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.191529 13245 ts_tablet_manager.cc:1367] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.120s user 0.006s sys 0.000s
W20240506 19:57:00.201864 12781 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:57:00.202067 12781 leader_election.cc:302] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.204814 14909 raft_consensus.cc:2726] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.205837 13245 raft_consensus.cc:348] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.206095 13245 raft_consensus.cc:374] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.206415 13245 raft_consensus.cc:729] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.206658 13245 consensus_queue.cc:260] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.207859 13245 ts_tablet_manager.cc:1398] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
W20240506 19:57:00.209388 13058 consensus_peers.cc:485] T 59b022c086d141a19a9b35c4a8c8bef9 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.210304 14797 raft_consensus.cc:482] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.210453 14797 raft_consensus.cc:504] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.211104 14797 leader_election.cc:288] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.214995 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:00.215538 12922 leader_election.cc:341] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.210978 13244 tablet_bootstrap.cc:492] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.212733 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.218308 13249 tablet_bootstrap.cc:492] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.217528 13244 ts_tablet_manager.cc:1367] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.025s
W20240506 19:57:00.218497 12921 leader_election.cc:341] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.218804 12921 leader_election.cc:302] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.219213 13244 raft_consensus.cc:348] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.219398 13244 raft_consensus.cc:374] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.219380 14797 raft_consensus.cc:2726] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.219455 13244 raft_consensus.cc:729] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.221922 14910 raft_consensus.cc:482] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.222259 14910 raft_consensus.cc:504] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.223549 14910 leader_election.cc:288] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.223816 12783 leader_election.cc:341] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 31c440555cfe4195b3ae86e1f1eb6a34
I20240506 19:57:00.223937 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c440555cfe4195b3ae86e1f1eb6a34" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.224176 12987 raft_consensus.cc:2454] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:57:00.224304 13245 tablet_bootstrap.cc:492] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.223801 13249 ts_tablet_manager.cc:1367] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.132s user 0.001s sys 0.006s
I20240506 19:57:00.224587 12781 leader_election.cc:302] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.224987 14910 raft_consensus.cc:2781] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.225106 14910 raft_consensus.cc:482] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.225214 14910 raft_consensus.cc:3037] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.225160 13249 raft_consensus.cc:348] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.225368 13249 raft_consensus.cc:374] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.225431 13249 raft_consensus.cc:729] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.225641 13249 consensus_queue.cc:260] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.226219 13249 ts_tablet_manager.cc:1398] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:00.219651 13244 consensus_queue.cc:260] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.226691 13244 ts_tablet_manager.cc:1398] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240506 19:57:00.228058 13245 tablet_bootstrap.cc:654] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.228953 13244 tablet_bootstrap.cc:492] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.231343 13244 tablet_bootstrap.cc:654] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.232770 14910 raft_consensus.cc:504] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.245029 13249 tablet_bootstrap.cc:492] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:57:00.246747 12783 leader_election.cc:341] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 31c440555cfe4195b3ae86e1f1eb6a34
I20240506 19:57:00.249969 14915 raft_consensus.cc:482] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.250169 14915 raft_consensus.cc:504] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.250797 14915 leader_election.cc:288] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.251602 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6089f4f68bfe415e8d97dfd6849b7714" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.251816 12987 raft_consensus.cc:2454] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:00.252198 13057 leader_election.cc:302] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.252535 14915 raft_consensus.cc:2781] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.252635 14915 raft_consensus.cc:482] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.252746 14915 raft_consensus.cc:3037] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.252913 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6089f4f68bfe415e8d97dfd6849b7714" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.255108 14910 leader_election.cc:288] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.255487 14915 raft_consensus.cc:504] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.256104 14915 leader_election.cc:288] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.256522 13058 leader_election.cc:341] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.257012 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c440555cfe4195b3ae86e1f1eb6a34" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.257184 12987 raft_consensus.cc:3037] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.257838 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6089f4f68bfe415e8d97dfd6849b7714" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:00.259816 12987 raft_consensus.cc:2454] T 31c440555cfe4195b3ae86e1f1eb6a34 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:57:00.260247 12781 leader_election.cc:302] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:57:00.260419 13058 leader_election.cc:341] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.261145 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6089f4f68bfe415e8d97dfd6849b7714" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.261399 12987 raft_consensus.cc:3037] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.262073 14910 raft_consensus.cc:2781] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.263500 14910 raft_consensus.cc:686] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:57:00.263810 14910 consensus_queue.cc:237] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.265254 14915 consensus_queue.cc:1035] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.266220 12987 raft_consensus.cc:2454] T 6089f4f68bfe415e8d97dfd6849b7714 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.266664 13057 leader_election.cc:302] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.267163 14915 raft_consensus.cc:2781] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.267349 14915 raft_consensus.cc:686] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.267688 14915 consensus_queue.cc:237] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.269227 13249 tablet_bootstrap.cc:654] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.269343 13245 tablet_bootstrap.cc:492] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.269497 13245 ts_tablet_manager.cc:1367] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240506 19:57:00.270188 13245 raft_consensus.cc:348] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.270439 13245 raft_consensus.cc:374] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.270522 13245 raft_consensus.cc:729] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.270773 13245 consensus_queue.cc:260] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.273726 14656 consensus_queue.cc:1035] T ff3049a9fa8049e5b84230e5e6e8245c P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.274600 13245 ts_tablet_manager.cc:1398] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:00.276481 13245 tablet_bootstrap.cc:492] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.277606 14909 raft_consensus.cc:482] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.277788 14909 raft_consensus.cc:504] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.278167 13245 tablet_bootstrap.cc:654] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.278636 14909 leader_election.cc:288] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.279819 12783 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 0aa8f9aaca444c839a88c07f58cb6ad6
W20240506 19:57:00.280458 12781 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:57:00.280640 12781 leader_election.cc:302] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.280897 14909 raft_consensus.cc:2726] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.286154 14797 raft_consensus.cc:482] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.286301 14797 raft_consensus.cc:504] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.286939 14797 leader_election.cc:288] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.287863 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:00.288506 12922 leader_election.cc:341] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.289327 14909 raft_consensus.cc:482] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.289498 14909 raft_consensus.cc:504] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.289901 13245 tablet_bootstrap.cc:492] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.290170 14909 leader_election.cc:288] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:00.290807 12783 leader_election.cc:341] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bece304e0ba048ecbb2d92aa71fdf707
I20240506 19:57:00.290933 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.290920 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bece304e0ba048ecbb2d92aa71fdf707" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.291016 13245 ts_tablet_manager.cc:1367] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240506 19:57:00.291129 12982 raft_consensus.cc:2454] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
W20240506 19:57:00.291630 12921 leader_election.cc:341] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.291740 12921 leader_election.cc:302] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.292135 14797 raft_consensus.cc:2726] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.292023 13245 raft_consensus.cc:348] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.292356 13245 raft_consensus.cc:374] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.292471 13245 raft_consensus.cc:729] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.292482 12781 leader_election.cc:302] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.292877 14909 raft_consensus.cc:2781] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.292810 13245 consensus_queue.cc:260] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.292977 14909 raft_consensus.cc:482] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.293064 14909 raft_consensus.cc:3037] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.293512 13245 ts_tablet_manager.cc:1398] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:00.295214 13245 tablet_bootstrap.cc:492] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.296190 14909 raft_consensus.cc:504] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.297012 13245 tablet_bootstrap.cc:654] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:00.297739 12783 leader_election.cc:341] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bece304e0ba048ecbb2d92aa71fdf707
I20240506 19:57:00.298115 14909 leader_election.cc:288] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.298945 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bece304e0ba048ecbb2d92aa71fdf707" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.299139 12982 raft_consensus.cc:3037] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.302210 12982 raft_consensus.cc:2454] T bece304e0ba048ecbb2d92aa71fdf707 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:57:00.302798 12781 leader_election.cc:302] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.304412 14909 raft_consensus.cc:2781] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.304562 14909 raft_consensus.cc:686] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:57:00.308259 13249 tablet_bootstrap.cc:492] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.308696 13249 ts_tablet_manager.cc:1367] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.003s
I20240506 19:57:00.306025 14909 consensus_queue.cc:237] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.309959 14797 raft_consensus.cc:482] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.310885 14797 raft_consensus.cc:504] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.311558 14797 leader_election.cc:288] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:00.312104 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986f9c83b04f41b49e9b78fce731a90f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.312351 13120 raft_consensus.cc:2454] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:00.312582 13249 raft_consensus.cc:348] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.312749 13249 raft_consensus.cc:374] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.312804 13249 raft_consensus.cc:729] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.312952 12922 leader_election.cc:302] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.313321 14797 raft_consensus.cc:2781] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.313407 14797 raft_consensus.cc:482] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.313469 14797 raft_consensus.cc:3037] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.313824 13244 tablet_bootstrap.cc:492] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.313951 13244 ts_tablet_manager.cc:1367] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.006s
I20240506 19:57:00.314677 13244 raft_consensus.cc:348] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.314862 13244 raft_consensus.cc:374] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.314908 13244 raft_consensus.cc:729] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.315376 13249 consensus_queue.cc:260] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.316053 14797 raft_consensus.cc:504] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.316650 14797 leader_election.cc:288] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:00.317683 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986f9c83b04f41b49e9b78fce731a90f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:00.317878 13120 raft_consensus.cc:3037] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.320911 13120 raft_consensus.cc:2454] T 986f9c83b04f41b49e9b78fce731a90f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:00.321465 12922 leader_election.cc:302] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.321584 14700 raft_consensus.cc:482] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.321682 14700 raft_consensus.cc:504] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.321815 14739 raft_consensus.cc:2781] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.321913 14739 raft_consensus.cc:686] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:00.322299 14700 leader_election.cc:288] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.322163 14739 consensus_queue.cc:237] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.316061 13249 ts_tablet_manager.cc:1398] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240506 19:57:00.315109 13244 consensus_queue.cc:260] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.324182 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028055b9df1f4667bd2e54909690e327" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:00.324717 12922 leader_election.cc:341] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.331802 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986f9c83b04f41b49e9b78fce731a90f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.332584 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986f9c83b04f41b49e9b78fce731a90f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:00.332820 13249 tablet_bootstrap.cc:492] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:00.333475 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028055b9df1f4667bd2e54909690e327" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.334254 12921 leader_election.cc:341] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:00.334798 12921 leader_election.cc:341] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:00.335557 12921 leader_election.cc:341] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.335649 12921 leader_election.cc:302] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.336000 14739 raft_consensus.cc:2726] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:00.336546 12781 consensus_peers.cc:485] T 40b9731a473449b19a9e95653a9d0830 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Couldn't send request to peer 059a5c4a3f7e49ef95af4b8341702745. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40b9731a473449b19a9e95653a9d0830. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:00.337046 13058 consensus_peers.cc:485] T e63baf59dc8244daad831b2987a51cfb P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.337867 14785 consensus_queue.cc:1035] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.336000 13249 tablet_bootstrap.cc:654] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.338194 13244 ts_tablet_manager.cc:1398] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20240506 19:57:00.338487 13244 tablet_bootstrap.cc:492] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.340483 13244 tablet_bootstrap.cc:654] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.343605 13245 tablet_bootstrap.cc:492] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.343773 13245 ts_tablet_manager.cc:1367] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.000s
I20240506 19:57:00.344666 13245 raft_consensus.cc:348] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.344882 13245 raft_consensus.cc:374] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.344964 13245 raft_consensus.cc:729] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.346130 13245 consensus_queue.cc:260] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.348074 13245 ts_tablet_manager.cc:1398] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:00.348374 13245 tablet_bootstrap.cc:492] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:00.358983 12921 consensus_peers.cc:485] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.359488 13244 tablet_bootstrap.cc:492] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.359637 13244 ts_tablet_manager.cc:1367] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20240506 19:57:00.360380 13244 raft_consensus.cc:348] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.360589 13244 raft_consensus.cc:374] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.360646 13244 raft_consensus.cc:729] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.361037 13244 consensus_queue.cc:260] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.361742 13244 ts_tablet_manager.cc:1398] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:00.362016 13244 tablet_bootstrap.cc:492] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.363620 13244 tablet_bootstrap.cc:654] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.365726 13249 tablet_bootstrap.cc:492] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.365859 13249 ts_tablet_manager.cc:1367] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240506 19:57:00.366763 13249 raft_consensus.cc:348] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.366961 13249 raft_consensus.cc:374] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.367028 13249 raft_consensus.cc:729] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:57:00.366755 13058 consensus_peers.cc:485] T 8d09eafa82a0423a8358958e78f728e2 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.353266 13245 tablet_bootstrap.cc:654] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.371337 13249 consensus_queue.cc:260] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.375161 13249 ts_tablet_manager.cc:1398] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:57:00.376271 13249 tablet_bootstrap.cc:492] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:00.383484 13249 tablet_bootstrap.cc:654] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.389560 13244 tablet_bootstrap.cc:492] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.389727 13244 ts_tablet_manager.cc:1367] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20240506 19:57:00.390545 13244 raft_consensus.cc:348] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.390863 13244 raft_consensus.cc:374] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.390916 13244 raft_consensus.cc:729] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.391209 13244 consensus_queue.cc:260] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.393357 13244 ts_tablet_manager.cc:1398] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240506 19:57:00.396520 13244 tablet_bootstrap.cc:492] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.396891 13245 tablet_bootstrap.cc:492] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.397100 13245 ts_tablet_manager.cc:1367] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.006s
I20240506 19:57:00.397955 13245 raft_consensus.cc:348] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.399437 13245 raft_consensus.cc:374] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.399605 13245 raft_consensus.cc:729] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.399894 13245 consensus_queue.cc:260] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.400490 13245 ts_tablet_manager.cc:1398] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 19:57:00.401130 13245 tablet_bootstrap.cc:492] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.407037 13245 tablet_bootstrap.cc:654] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.411206 13244 tablet_bootstrap.cc:654] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.414594 13249 tablet_bootstrap.cc:492] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.414753 13249 ts_tablet_manager.cc:1367] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240506 19:57:00.415673 13249 raft_consensus.cc:348] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.415866 13249 raft_consensus.cc:374] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.415917 13249 raft_consensus.cc:729] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.416128 13249 consensus_queue.cc:260] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.416718 13249 ts_tablet_manager.cc:1398] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:00.416989 13249 tablet_bootstrap.cc:492] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:00.421195 13249 tablet_bootstrap.cc:654] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.422336 14739 consensus_queue.cc:1035] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.426215 14739 consensus_queue.cc:1035] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.429469 14915 raft_consensus.cc:482] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.429662 14915 raft_consensus.cc:504] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.430415 14915 leader_election.cc:288] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.431697 13244 tablet_bootstrap.cc:492] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.431844 13244 ts_tablet_manager.cc:1367] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
W20240506 19:57:00.432219 12921 consensus_peers.cc:485] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.432665 13244 raft_consensus.cc:348] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.432876 13244 raft_consensus.cc:374] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.432940 13244 raft_consensus.cc:729] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.433051 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64485c3c13e94055a019f2ef7be1f2fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.433096 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64485c3c13e94055a019f2ef7be1f2fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.433144 13244 consensus_queue.cc:260] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.433233 12986 raft_consensus.cc:2454] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:57:00.433758 13058 leader_election.cc:341] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.434422 13057 leader_election.cc:302] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.435178 14915 raft_consensus.cc:2781] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.435254 14915 raft_consensus.cc:482] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.435312 14915 raft_consensus.cc:3037] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.437978 14915 raft_consensus.cc:504] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.439069 14915 leader_election.cc:288] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.439415 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64485c3c13e94055a019f2ef7be1f2fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:00.439635 14797 consensus_queue.cc:1035] T b625c2bbd89746bc90b8c4c2483b3741 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:57:00.439985 13058 leader_election.cc:341] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.440141 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64485c3c13e94055a019f2ef7be1f2fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.440305 12986 raft_consensus.cc:3037] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.443835 12986 raft_consensus.cc:2454] T 64485c3c13e94055a019f2ef7be1f2fe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.444327 13057 leader_election.cc:302] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.444598 14915 raft_consensus.cc:2781] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.445009 14915 raft_consensus.cc:686] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.445364 14915 consensus_queue.cc:237] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.446604 13244 ts_tablet_manager.cc:1398] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240506 19:57:00.446918 13244 tablet_bootstrap.cc:492] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:00.451779 12921 consensus_peers.cc:485] T cd6245e4adce4038ad6f988877d75045 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.448953 14904 raft_consensus.cc:482] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.456738 13244 tablet_bootstrap.cc:654] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.456621 14904 raft_consensus.cc:504] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.457265 14904 leader_election.cc:288] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.457990 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e6f7555dfa4122ac00bae7b9e733fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.458205 12986 raft_consensus.cc:2454] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:00.458593 13057 leader_election.cc:302] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.459231 14904 raft_consensus.cc:2781] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.459307 14904 raft_consensus.cc:482] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.459367 14904 raft_consensus.cc:3037] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.463362 14904 raft_consensus.cc:504] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.465081 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e6f7555dfa4122ac00bae7b9e733fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.465343 12986 raft_consensus.cc:3037] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.471405 12986 raft_consensus.cc:2454] T 76e6f7555dfa4122ac00bae7b9e733fe P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.471958 13057 leader_election.cc:302] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.472327 14656 raft_consensus.cc:2781] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.472455 14656 raft_consensus.cc:686] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.472766 14656 consensus_queue.cc:237] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.474555 14904 leader_election.cc:288] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.483022 13245 tablet_bootstrap.cc:492] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.483171 13245 ts_tablet_manager.cc:1367] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.082s user 0.022s sys 0.001s
I20240506 19:57:00.484648 13245 raft_consensus.cc:348] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.484846 13245 raft_consensus.cc:374] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.484894 13245 raft_consensus.cc:729] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.485131 13245 consensus_queue.cc:260] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.486711 13245 ts_tablet_manager.cc:1398] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:00.481921 13249 tablet_bootstrap.cc:492] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.487555 13249 ts_tablet_manager.cc:1367] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20240506 19:57:00.489531 13245 tablet_bootstrap.cc:492] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.491065 13245 tablet_bootstrap.cc:654] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.490751 13249 raft_consensus.cc:348] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.492525 13249 raft_consensus.cc:374] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.492861 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e6f7555dfa4122ac00bae7b9e733fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.493650 13058 leader_election.cc:341] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.494953 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e6f7555dfa4122ac00bae7b9e733fe" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.495517 13058 leader_election.cc:341] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.501801 13245 tablet_bootstrap.cc:492] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.502132 13245 ts_tablet_manager.cc:1367] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240506 19:57:00.502805 13245 raft_consensus.cc:348] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.503082 13245 raft_consensus.cc:374] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.503190 13245 raft_consensus.cc:729] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.503473 13245 consensus_queue.cc:260] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.504948 13245 ts_tablet_manager.cc:1398] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:57:00.506125 13245 tablet_bootstrap.cc:492] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.507661 13245 tablet_bootstrap.cc:654] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.507756 13244 tablet_bootstrap.cc:492] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.507980 13244 ts_tablet_manager.cc:1367] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.003s
I20240506 19:57:00.509022 13244 raft_consensus.cc:348] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.509341 13244 raft_consensus.cc:374] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.509420 13244 raft_consensus.cc:729] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.509779 13244 consensus_queue.cc:260] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.511090 13244 ts_tablet_manager.cc:1398] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:57:00.515794 13244 tablet_bootstrap.cc:492] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:00.520156 13058 consensus_peers.cc:485] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.525429 13245 tablet_bootstrap.cc:492] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.525646 13245 ts_tablet_manager.cc:1367] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20240506 19:57:00.526444 13245 raft_consensus.cc:348] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.526717 13245 raft_consensus.cc:374] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.526823 13245 raft_consensus.cc:729] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.528272 13244 tablet_bootstrap.cc:654] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.538439 13244 tablet_bootstrap.cc:492] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.538622 13244 ts_tablet_manager.cc:1367] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20240506 19:57:00.538590 13245 consensus_queue.cc:260] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.539323 13244 raft_consensus.cc:348] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.539525 13244 raft_consensus.cc:374] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.539598 13244 raft_consensus.cc:729] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.539856 13244 consensus_queue.cc:260] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.543561 13245 ts_tablet_manager.cc:1398] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240506 19:57:00.544994 13245 tablet_bootstrap.cc:492] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.554622 13249 raft_consensus.cc:729] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.555001 13249 consensus_queue.cc:260] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.555945 13249 ts_tablet_manager.cc:1398] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.068s user 0.023s sys 0.000s
I20240506 19:57:00.556221 13244 ts_tablet_manager.cc:1398] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240506 19:57:00.555498 13245 tablet_bootstrap.cc:654] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.556524 14915 raft_consensus.cc:482] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.556669 14915 raft_consensus.cc:504] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.557314 14915 leader_election.cc:288] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.558132 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.558586 14797 raft_consensus.cc:482] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.558705 14797 raft_consensus.cc:504] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.559258 14797 leader_election.cc:288] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.559693 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.559885 12986 raft_consensus.cc:2454] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:00.560434 13057 leader_election.cc:302] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.560765 14915 raft_consensus.cc:2781] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.560833 14915 raft_consensus.cc:482] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.560912 14915 raft_consensus.cc:3037] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.562570 13244 tablet_bootstrap.cc:492] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.563104 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de26793c006440dc9aecc07963ddb85e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.563278 13114 raft_consensus.cc:2454] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:00.563627 14915 raft_consensus.cc:504] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.563774 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de26793c006440dc9aecc07963ddb85e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.564265 13244 tablet_bootstrap.cc:654] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:00.564376 12921 leader_election.cc:341] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.564949 12922 leader_election.cc:302] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.565776 14739 raft_consensus.cc:2781] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.565845 14739 raft_consensus.cc:482] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.565933 14739 raft_consensus.cc:3037] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.567977 14915 leader_election.cc:288] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.568423 13249 tablet_bootstrap.cc:492] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:57:00.570084 13058 leader_election.cc:341] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.570439 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.570981 13058 leader_election.cc:341] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.573273 14910 raft_consensus.cc:482] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.573468 14910 raft_consensus.cc:504] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.568393 14739 raft_consensus.cc:504] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.574141 14910 leader_election.cc:288] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.574156 14739 leader_election.cc:288] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.568830 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.574326 12986 raft_consensus.cc:3037] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.574720 13249 tablet_bootstrap.cc:654] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:00.575464 12783 leader_election.cc:341] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1750c4dd8cdc40509c553b63d166c466
I20240506 19:57:00.577016 12986 raft_consensus.cc:2454] T 9fed2f12eda54be0879b1e9c33ca3e31 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.578055 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de26793c006440dc9aecc07963ddb85e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.578647 12921 leader_election.cc:341] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.579255 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de26793c006440dc9aecc07963ddb85e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:00.579340 13057 leader_election.cc:302] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.579439 13114 raft_consensus.cc:3037] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.579815 14915 raft_consensus.cc:2781] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.579972 14915 raft_consensus.cc:686] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.580288 14656 raft_consensus.cc:482] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.580248 14915 consensus_queue.cc:237] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.580379 14656 raft_consensus.cc:504] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.580873 14656 leader_election.cc:288] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.581544 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:00.581715 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.581905 12987 raft_consensus.cc:2454] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:57:00.582043 13058 leader_election.cc:341] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.582229 13114 raft_consensus.cc:2454] T de26793c006440dc9aecc07963ddb85e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:00.582329 13057 leader_election.cc:302] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.582562 14656 raft_consensus.cc:2781] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.582633 14656 raft_consensus.cc:482] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.582696 14656 raft_consensus.cc:3037] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.582795 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1750c4dd8cdc40509c553b63d166c466" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.584244 13244 tablet_bootstrap.cc:492] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.584399 13244 ts_tablet_manager.cc:1367] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240506 19:57:00.585137 13244 raft_consensus.cc:348] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.585129 14656 raft_consensus.cc:504] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.585359 13244 raft_consensus.cc:374] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.585429 13244 raft_consensus.cc:729] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.585839 13244 consensus_queue.cc:260] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.586602 12922 leader_election.cc:302] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.588882 12781 leader_election.cc:302] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:00.589318 14910 raft_consensus.cc:2726] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.589471 13244 ts_tablet_manager.cc:1398] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:00.589794 13244 tablet_bootstrap.cc:492] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.589882 14797 raft_consensus.cc:2781] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.590020 14797 raft_consensus.cc:686] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:00.590265 14797 consensus_queue.cc:237] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.591553 13244 tablet_bootstrap.cc:654] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.593797 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.594851 13058 leader_election.cc:341] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.595911 14656 leader_election.cc:288] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.596470 14739 raft_consensus.cc:482] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.596917 14739 raft_consensus.cc:504] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.598800 14739 leader_election.cc:288] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.599081 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cade6b1c5a034b23aa33b2349e1dab95" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.599658 12921 leader_election.cc:341] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.599839 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.599823 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cade6b1c5a034b23aa33b2349e1dab95" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.600071 12986 raft_consensus.cc:3037] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.600215 13114 raft_consensus.cc:2454] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:00.604255 12986 raft_consensus.cc:2454] T df160fd5d8ce4bc192913ce702f8f478 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.605362 12922 leader_election.cc:302] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.606249 13057 leader_election.cc:302] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.606611 14656 raft_consensus.cc:2781] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.606822 14739 raft_consensus.cc:2781] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.606901 14739 raft_consensus.cc:482] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.606977 14739 raft_consensus.cc:3037] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.607201 14656 raft_consensus.cc:686] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.607506 14656 consensus_queue.cc:237] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.609879 14739 raft_consensus.cc:504] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.610949 14739 leader_election.cc:288] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.611335 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cade6b1c5a034b23aa33b2349e1dab95" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:00.612494 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cade6b1c5a034b23aa33b2349e1dab95" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:00.612692 13114 raft_consensus.cc:3037] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:00.613416 12921 leader_election.cc:341] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.615324 13114 raft_consensus.cc:2454] T cade6b1c5a034b23aa33b2349e1dab95 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:00.617549 12922 leader_election.cc:302] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.620200 14739 raft_consensus.cc:2781] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.620484 14739 raft_consensus.cc:686] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:00.620798 14739 consensus_queue.cc:237] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.628265 12845 raft_consensus.cc:3037] T f759b560e2ab41a6a076976ccca811d1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.629016 13245 tablet_bootstrap.cc:492] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.629148 13245 ts_tablet_manager.cc:1367] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.000s
I20240506 19:57:00.629786 14797 raft_consensus.cc:482] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.629930 14797 raft_consensus.cc:504] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.630353 13245 raft_consensus.cc:348] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.630532 13245 raft_consensus.cc:374] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.630527 13244 tablet_bootstrap.cc:492] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.630578 13245 raft_consensus.cc:729] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.630795 13245 consensus_queue.cc:260] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.631330 13245 ts_tablet_manager.cc:1398] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:00.631405 13244 ts_tablet_manager.cc:1367] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.011s
I20240506 19:57:00.631630 13245 tablet_bootstrap.cc:492] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.634342 13244 raft_consensus.cc:348] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.635473 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9f32618b64ffa92b5a868081c36a6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.635684 13114 raft_consensus.cc:2454] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:00.636085 12922 leader_election.cc:302] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.636375 13244 raft_consensus.cc:374] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.636507 13244 raft_consensus.cc:729] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.636740 13244 consensus_queue.cc:260] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.636876 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9f32618b64ffa92b5a868081c36a6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.637341 12921 leader_election.cc:341] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.637398 14784 raft_consensus.cc:2781] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.637486 14784 raft_consensus.cc:482] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.637566 14784 raft_consensus.cc:3037] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.638227 13244 ts_tablet_manager.cc:1398] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:57:00.640228 14784 raft_consensus.cc:504] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.641108 14797 leader_election.cc:288] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.641147 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9f32618b64ffa92b5a868081c36a6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:00.641351 13114 raft_consensus.cc:3037] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.641660 14784 leader_election.cc:288] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.643985 13114 raft_consensus.cc:2454] T 55d9f32618b64ffa92b5a868081c36a6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:00.644080 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9f32618b64ffa92b5a868081c36a6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.644747 12921 leader_election.cc:341] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.645471 13244 tablet_bootstrap.cc:492] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.647519 13244 tablet_bootstrap.cc:654] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.650601 13244 tablet_bootstrap.cc:492] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.650804 13244 ts_tablet_manager.cc:1367] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240506 19:57:00.651743 13244 raft_consensus.cc:348] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.651960 13244 raft_consensus.cc:374] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.652038 13244 raft_consensus.cc:729] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.653110 12922 leader_election.cc:302] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.653503 14784 raft_consensus.cc:2781] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.654960 14784 raft_consensus.cc:686] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:00.655239 14784 consensus_queue.cc:237] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.663024 14700 raft_consensus.cc:482] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.663197 14700 raft_consensus.cc:504] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.663895 14700 leader_election.cc:288] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.664676 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a8531a1793404699532295164a3895" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.665220 12921 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:00.665673 12922 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f3a8531a1793404699532295164a3895
I20240506 19:57:00.665776 12922 leader_election.cc:302] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.666051 14700 raft_consensus.cc:2726] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.676265 13245 tablet_bootstrap.cc:654] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:00.678061 14859 op_tracker.cc:283] OpTracker waiting for 2 outstanding ops to complete now for 1.030s
I20240506 19:57:00.678189 14859 op_tracker.cc:285] Dumping up to 50 currently running ops:
I20240506 19:57:00.678254 14859 op_tracker.cc:289] R-P WriteOp [type=FOLLOWER, start_time=2024-05-06 19:56:58, state=WriteOpState 0x617000182c80 [op_id=(term: 1 index: 1445), ts=7024744115520880640, rows=[MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>, MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>, MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>, ...]]]
I20240506 19:57:00.678716 14859 op_tracker.cc:289] R-P WriteOp [type=FOLLOWER, start_time=2024-05-06 19:56:58, state=WriteOpState 0x617000182900 [op_id=(term: 1 index: 1444), ts=7024744115348185088, rows=[MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
I20240506 19:57:00.681427 13244 consensus_queue.cc:260] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.682334 13244 ts_tablet_manager.cc:1398] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20240506 19:57:00.684540 13244 tablet_bootstrap.cc:492] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.686839 13244 tablet_bootstrap.cc:654] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.688457 13249 tablet_bootstrap.cc:492] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.688601 13249 ts_tablet_manager.cc:1367] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.000s
I20240506 19:57:00.689276 13249 raft_consensus.cc:348] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.689505 13249 raft_consensus.cc:374] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.689566 13249 raft_consensus.cc:729] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.689787 13249 consensus_queue.cc:260] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.690305 13249 ts_tablet_manager.cc:1398] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:57:00.690579 13249 tablet_bootstrap.cc:492] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:00.692117 13249 tablet_bootstrap.cc:654] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.697636 14656 consensus_queue.cc:1035] T ecc88443c04b42d19cc7fba339eeeaf5 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:00.704717 13245 tablet_bootstrap.cc:492] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.704926 13245 ts_tablet_manager.cc:1367] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.000s
I20240506 19:57:00.706101 13245 raft_consensus.cc:348] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.706379 13245 raft_consensus.cc:374] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.706463 13245 raft_consensus.cc:729] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.706760 13245 consensus_queue.cc:260] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.707540 13245 ts_tablet_manager.cc:1398] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:00.718617 13245 tablet_bootstrap.cc:492] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.720309 13245 tablet_bootstrap.cc:654] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.724102 12845 raft_consensus.cc:3037] T 458314e4653748e48cf646c72373fc3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.730577 14901 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:57:00.738667 12483 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
/tmp/dist-test-taskQMIkBc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.254:33495
--webserver_interface=127.12.48.254
--webserver_port=40249
--builtin_ntp_servers=127.12.48.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 19:57:00.742069 14917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d4482cd59a4e5687992b1f8a177373. Latest consensus state: current_term: 2 leader_uuid: "27d4482cd59a4e5687992b1f8a177373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:57:00.742383 14917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:57:00.755419 14965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "27d4482cd59a4e5687992b1f8a177373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:57:00.755709 14965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 [sys.catalog]: This master's current role is: LEADER
I20240506 19:57:00.769968 14917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "27d4482cd59a4e5687992b1f8a177373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } } }
I20240506 19:57:00.770326 14917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 19:57:00.806704 14784 raft_consensus.cc:482] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.806955 14784 raft_consensus.cc:504] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:00.808374 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ce8749846847108edf5f554139cbd9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.809646 12921 leader_election.cc:341] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.814128 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ce8749846847108edf5f554139cbd9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:00.814296 14784 leader_election.cc:288] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:57:00.814642 12922 leader_election.cc:341] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.814764 12922 leader_election.cc:302] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.816162 14739 raft_consensus.cc:2726] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.816316 14739 raft_consensus.cc:482] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.816421 14739 raft_consensus.cc:504] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.817080 14739 leader_election.cc:288] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.817844 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.818418 12921 leader_election.cc:341] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.819038 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:00.820256 12922 leader_election.cc:341] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.820366 12922 leader_election.cc:302] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.820868 14700 raft_consensus.cc:2726] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.823360 12842 raft_consensus.cc:3037] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.853139 14987 tablet.cc:1722] T 9d0d827d732e4261a7a8edb549c1c8e3 P ef18e88b87064ed4b6c20f9325ebcc33: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240506 19:57:00.857492 14739 raft_consensus.cc:482] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.857671 14739 raft_consensus.cc:504] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.859494 14739 leader_election.cc:288] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:00.860263 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43da1d12ce647dcbcd31263932b2f54" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.862177 12921 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:00.865357 12922 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d43da1d12ce647dcbcd31263932b2f54
I20240506 19:57:00.865496 12922 leader_election.cc:302] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.869405 14739 raft_consensus.cc:2726] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:00.890692 13249 tablet_bootstrap.cc:492] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:00.892532 13249 ts_tablet_manager.cc:1367] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.202s user 0.007s sys 0.003s
I20240506 19:57:00.894565 14915 raft_consensus.cc:482] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:00.894790 14915 raft_consensus.cc:504] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.895537 14915 leader_election.cc:288] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.901417 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028055b9df1f4667bd2e54909690e327" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:00.902197 13058 leader_election.cc:341] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.902746 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028055b9df1f4667bd2e54909690e327" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:00.903034 12987 raft_consensus.cc:2454] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:00.903414 13057 leader_election.cc:302] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:00.903828 14915 raft_consensus.cc:2781] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:00.903896 14915 raft_consensus.cc:482] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:00.903949 14915 raft_consensus.cc:3037] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.907022 14915 raft_consensus.cc:504] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.907883 14915 leader_election.cc:288] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:00.908943 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028055b9df1f4667bd2e54909690e327" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:00.909198 12986 raft_consensus.cc:3037] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.912503 12986 raft_consensus.cc:2454] T 028055b9df1f4667bd2e54909690e327 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:00.915966 13057 leader_election.cc:302] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:00.916738 14915 raft_consensus.cc:2781] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:00.916980 14915 raft_consensus.cc:686] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:00.917841 12848 raft_consensus.cc:3037] T 1750c4dd8cdc40509c553b63d166c466 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:00.917274 14915 consensus_queue.cc:237] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.925067 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028055b9df1f4667bd2e54909690e327" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:00.926028 13058 leader_election.cc:341] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:00.930675 14859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 19:57:00.934862 13249 raft_consensus.cc:348] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.935128 13249 raft_consensus.cc:374] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.935186 13249 raft_consensus.cc:729] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.935231 13244 tablet_bootstrap.cc:492] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.935384 13244 ts_tablet_manager.cc:1367] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.251s user 0.023s sys 0.023s
I20240506 19:57:00.935591 13249 consensus_queue.cc:260] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.936156 13244 raft_consensus.cc:348] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.936347 13244 raft_consensus.cc:374] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.936394 13244 raft_consensus.cc:729] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.936573 13244 consensus_queue.cc:260] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.937239 13244 ts_tablet_manager.cc:1398] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:00.937544 13244 tablet_bootstrap.cc:492] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.938938 13244 tablet_bootstrap.cc:654] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.939878 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6 [id=0090970ea0b54ee4a872766d8145bd6f]
W20240506 19:57:00.940336 13058 consensus_peers.cc:485] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.940897 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_8d7fd547dfb74b66a34d4207ef4bf800 [id=01881436327441a384925f571f99bf14]
I20240506 19:57:00.941421 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6fb4b0e1d5c94d88b7f9792005908344 [id=0206d0193f404359bf21bc2d8662c3c0]
I20240506 19:57:00.942153 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_16507180ad3142ee968513548406bd0b [id=02a9361327124ac9a5359df5020f59e4]
I20240506 19:57:00.943706 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b261bf09f51646a0877f068d9eb2ab36 [id=034d601564f94db89efafdf7d3b35f05]
I20240506 19:57:00.944520 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ecb75e9b5c5d4c4ebd6f3684a6a350f9 [id=034d7824127b4f94aa24bdeb76bfbd7c]
I20240506 19:57:00.947252 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1698ae51beba4bd0a7c376126987543b [id=049f48b0d93a41ca93371a76316f7e9d]
I20240506 19:57:00.949514 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_251bb78261d6402bbf113e3110a57ba2 [id=05680d808e924fcfbe8d79454a3ded17]
I20240506 19:57:00.952143 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_07bfadf060f9447e80912279571d2d43 [id=075c32cd3d9f4029be393b323b001186]
I20240506 19:57:00.956784 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e83de886956a4a2ba07ef464224044f6 [id=0769cf069b8847faab1e5922412d544c]
W20240506 19:57:00.957262 12921 consensus_peers.cc:485] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.959151 13244 tablet_bootstrap.cc:492] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:00.959453 13244 ts_tablet_manager.cc:1367] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240506 19:57:00.960775 13244 raft_consensus.cc:348] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.961169 13244 raft_consensus.cc:374] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.961385 13244 raft_consensus.cc:729] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.961796 13244 consensus_queue.cc:260] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.965760 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_30f6a4c5ddac420ca7fafd3366c7cd17 [id=077490f89fa147baa2713ab465729c7e]
I20240506 19:57:00.966976 13244 ts_tablet_manager.cc:1398] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240506 19:57:00.969769 13244 tablet_bootstrap.cc:492] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:00.971110 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0fba0d9aab8e488b80a10c9afe3e7aee [id=07f6d1a0d63746d0bef1830f74f365d7]
I20240506 19:57:00.973632 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6c86bd8e4c954a06a24af7379533208a [id=084f9d59425445c1aad3f9171f2c05d2]
I20240506 19:57:00.975848 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2da22bc9de82467f80578b673f6faab6 [id=08b553c14b6743b38fda0abacb84766d]
I20240506 19:57:00.982690 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a4962437b4904f948a5ceaf065c7ab3e [id=09e89c3a03164e878e6182dddb162d84]
I20240506 19:57:00.984927 13244 tablet_bootstrap.cc:654] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:00.988688 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f162120d90f640e09da676cac7d0c187 [id=0b05c2baa470418c9f1c32a7de9167da]
I20240506 19:57:00.991703 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_16db473ae5ed4bd0ab2c8795b490f5cf [id=0b3a486ff6d24c8795de3e53e83021a2]
W20240506 19:57:00.992413 13058 consensus_peers.cc:485] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:00.993590 13245 tablet_bootstrap.cc:492] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:00.993736 13245 ts_tablet_manager.cc:1367] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.275s user 0.028s sys 0.072s
I20240506 19:57:00.994488 13245 raft_consensus.cc:348] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.994748 13245 raft_consensus.cc:374] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:00.994804 13245 raft_consensus.cc:729] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:00.995013 13245 consensus_queue.cc:260] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:00.995510 13245 ts_tablet_manager.cc:1398] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:00.995820 13245 tablet_bootstrap.cc:492] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:00.996276 13249 ts_tablet_manager.cc:1398] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.104s user 0.002s sys 0.000s
I20240506 19:57:00.997628 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d630c009c25b457388c1041305861480 [id=0c5aabe04b194b59b5b6adcd7389744a]
I20240506 19:57:01.001257 13245 tablet_bootstrap.cc:654] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.003685 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_efa08afa30fd48009749a76c90bd6725 [id=0df30d7b42ff4869ad37a24a82300280]
I20240506 19:57:01.005882 13249 tablet_bootstrap.cc:492] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:01.010339 13244 tablet_bootstrap.cc:492] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.010636 13244 ts_tablet_manager.cc:1367] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.000s
I20240506 19:57:01.011814 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ef58202ae07041c18e495f8a92ec3eb5 [id=0efca22312684fd0ba5a7db69f4fed15]
I20240506 19:57:01.012008 13244 raft_consensus.cc:348] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.012398 13244 raft_consensus.cc:374] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.012523 13244 raft_consensus.cc:729] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.012948 13244 consensus_queue.cc:260] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.014760 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_5140ce0f4218408182d418c041a2535c [id=0f7c418d978f4813bf5db9a9e31bc24d]
I20240506 19:57:01.016278 13244 ts_tablet_manager.cc:1398] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:57:01.017021 14910 raft_consensus.cc:482] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.017160 14910 raft_consensus.cc:504] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.017944 14910 leader_election.cc:288] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.019399 13244 tablet_bootstrap.cc:492] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:01.020413 12783 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:57:01.022033 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_51d50ac0d42d4388a250ec5fd84128a8 [id=0f8c1a749dd94feb907921bc2524dbc2]
I20240506 19:57:01.026320 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2a2d910a263840e7934d2972c9cd8a87 [id=0f8f012e7cf24efe9c4a08abf3fab067]
I20240506 19:57:01.029140 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_21676c5997464e2d8e94bb0efdfead41 [id=106b2ba1a92643d4892c5e9197d81117]
I20240506 19:57:01.031469 14915 consensus_queue.cc:1035] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.033612 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e87268955e8247709b420f45ae2e4e5d [id=12a8e13309bc412ba8404129577b3444]
I20240506 19:57:01.033550 13244 tablet_bootstrap.cc:654] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.039348 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0921ada34d224bd7b8bc59edc8627549 [id=14b3734b28354d60b48fa59014220a25]
W20240506 19:57:01.041419 12781 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:57:01.041613 12781 leader_election.cc:302] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.042244 14910 raft_consensus.cc:2726] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.043640 14915 consensus_queue.cc:1035] T 85365758f6a64536ab61e8349de63dea P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.045469 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7d21a9e7790049a286bd3224ce33ad10 [id=154b16b0c6b94668a30c453782899633]
I20240506 19:57:01.047189 12848 raft_consensus.cc:3037] T 4f382a89f94e4fdf9b8266d049635329 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.055411 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fa78b5478d7c4ca08cb6dcbb0b0b349e [id=15c9ba78bdf345b9b8838a4aeaebb885]
I20240506 19:57:01.056499 13244 tablet_bootstrap.cc:492] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:57:01.056712 13058 consensus_peers.cc:485] T 5fe8d4891c484627ac729971ff7aad3e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.056735 13244 ts_tablet_manager.cc:1367] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20240506 19:57:01.057507 14949 raft_consensus.cc:482] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.057637 14949 raft_consensus.cc:504] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.057996 13244 raft_consensus.cc:348] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.058266 14949 leader_election.cc:288] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.058348 13244 raft_consensus.cc:374] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.058457 13244 raft_consensus.cc:729] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.058810 13244 consensus_queue.cc:260] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.059478 13244 ts_tablet_manager.cc:1398] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240506 19:57:01.061056 12921 consensus_peers.cc:485] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.061020 13244 tablet_bootstrap.cc:492] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.061883 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b00381c6aa42c9b969410b51ebe81e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.062080 12986 raft_consensus.cc:2454] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:01.062057 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b00381c6aa42c9b969410b51ebe81e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.062428 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1d9621aac59e4d35ba7cf1d1d1206f64 [id=16258fb4a39c4effb6effa8e857b0b6f]
W20240506 19:57:01.063541 13058 leader_election.cc:341] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.063829 14784 consensus_queue.cc:1035] T a20016b56be94248905e0e8d6a35100f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.064491 13057 leader_election.cc:302] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.064946 14904 raft_consensus.cc:2781] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.065071 14904 raft_consensus.cc:482] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.065212 14904 raft_consensus.cc:3037] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.065961 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2]
I20240506 19:57:01.069029 14904 raft_consensus.cc:504] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.070173 14904 leader_election.cc:288] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.071556 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7b54e910804a4cbb8ec86b6e6f9a9246 [id=16f928ea6812485db743757c46b5348d]
I20240506 19:57:01.073129 13244 tablet_bootstrap.cc:654] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.075071 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4d9003ea123240c8806509c02e08ebe3 [id=17e3ad30ddd346be912bdb5ebfecdb7e]
I20240506 19:57:01.073514 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b00381c6aa42c9b969410b51ebe81e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:01.076287 13058 leader_election.cc:341] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.077524 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d7890a93b92a4c6d881598c0bb5f5a21 [id=17f3bc6b4b0645198e5a4d97d6b8a691]
I20240506 19:57:01.081488 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_946004ac61bd4421abf9cdfa64f35460 [id=1881dd13d96a4563a72c4791dcaccfca]
I20240506 19:57:01.084264 14915 raft_consensus.cc:482] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.084416 14915 raft_consensus.cc:504] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:01.084507 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b00381c6aa42c9b969410b51ebe81e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.084793 12986 raft_consensus.cc:3037] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.085064 14915 leader_election.cc:288] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.086215 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.086796 13058 leader_election.cc:341] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.087770 12986 raft_consensus.cc:2454] T 11b00381c6aa42c9b969410b51ebe81e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:01.088351 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eb945644f4344376aa5059f1a8223ce9 [id=18e06c57bec942d9b0c1339852f02dd1]
I20240506 19:57:01.092260 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1f6bae23516f4e79b0662b14c8a8ea3a [id=19bea1ba9dbf4b2787c397c6152b8c26]
I20240506 19:57:01.098073 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_dafad97b193f40038dfd516c93b79e96 [id=1a1d40c83b0744689cb8a85164109e0e]
I20240506 19:57:01.099596 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_85577a3ca1fb430194567dcaa70fa260 [id=1a7d71a5f5fe4f9e84c456f5ec8ea20a]
I20240506 19:57:01.106212 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1aacc41134ef4e40ac2c05d45c1faf3a [id=1a82494c8acb4e948297879f4b780060]
I20240506 19:57:01.096626 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.107844 12982 raft_consensus.cc:2454] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:57:01.098351 12921 consensus_peers.cc:485] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.108695 13057 leader_election.cc:302] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.109282 13057 leader_election.cc:302] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.109874 14915 raft_consensus.cc:2781] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.109951 14915 raft_consensus.cc:482] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.110041 14915 raft_consensus.cc:3037] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.110492 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4974044201774aa6adef60247d009bd7 [id=1bbd0af47d9a4dfb8f4d8e98117c888f]
I20240506 19:57:01.110862 13244 tablet_bootstrap.cc:492] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.110949 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a010c4cc7a5348d79e8c910419d72c98 [id=1ee0a9b9312f4deda04f6199c1014af5]
I20240506 19:57:01.111102 13244 ts_tablet_manager.cc:1367] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.000s
I20240506 19:57:01.111693 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f659f7ad515e4001891b15f486dc7b1a [id=1ef16dfbd249466fa6b0a7df43515b7b]
I20240506 19:57:01.112351 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e4fb89d4dd964179b1cd1c2f2c91567c [id=1f7e95047b4840499fa89ae4b887e279]
I20240506 19:57:01.112794 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6449ebda2e1b4b0088a1c4bd083cf123 [id=2126bfc39a3b4b34b6edf5460144fb46]
I20240506 19:57:01.113041 14915 raft_consensus.cc:504] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:01.113214 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a0d74db25f05423f8b5ae3b82c7a54b1 [id=21294b97805743eb8444823b07a0ed7b]
I20240506 19:57:01.114073 14915 leader_election.cc:288] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.114858 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.114604 13244 raft_consensus.cc:348] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.115172 13244 raft_consensus.cc:374] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.115520 13244 raft_consensus.cc:729] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.115846 14915 raft_consensus.cc:2781] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
W20240506 19:57:01.115556 13058 leader_election.cc:341] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.116600 14915 raft_consensus.cc:686] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:01.116909 14915 consensus_queue.cc:237] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.103318 13249 tablet_bootstrap.cc:654] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.118885 14739 raft_consensus.cc:482] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.119050 14739 raft_consensus.cc:504] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.115918 13244 consensus_queue.cc:260] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.120407 14739 leader_election.cc:288] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.120371 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b949c48806d45c3a74efbc8503c0bdb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.120913 12921 leader_election.cc:341] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.123809 12843 raft_consensus.cc:3037] T bff7050bbc2e47cf823d25ec10b89718 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.133672 13244 ts_tablet_manager.cc:1398] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240506 19:57:01.135845 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e74f7640674041f292cd9a7b4ffb3e9e [id=22080408cdb6434b89a4834e857cf550]
I20240506 19:57:01.136878 13244 tablet_bootstrap.cc:492] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:01.139299 13442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.254:33495, user_credentials={real_user=slave}} blocked reactor thread for 436301us
I20240506 19:57:01.143980 13245 tablet_bootstrap.cc:492] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.144146 13245 ts_tablet_manager.cc:1367] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.148s user 0.019s sys 0.017s
I20240506 19:57:01.145511 14910 raft_consensus.cc:482] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.145714 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1d4371bd92c34570ba9ffca5ae2a387d [id=224d462b1c6d4535937a0d82ff847771]
I20240506 19:57:01.145687 14910 raft_consensus.cc:504] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.144932 13245 raft_consensus.cc:348] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.146289 13245 raft_consensus.cc:374] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.146313 14910 leader_election.cc:288] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.146354 13245 raft_consensus.cc:729] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.147276 13245 consensus_queue.cc:260] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.147846 13245 ts_tablet_manager.cc:1398] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:57:01.148239 14915 raft_consensus.cc:482] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.148416 14915 raft_consensus.cc:504] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.149518 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb9c05d703f417780e77359e19e542e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.150161 14915 leader_election.cc:288] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:01.152403 13058 leader_election.cc:341] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.153512 13244 tablet_bootstrap.cc:654] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.157157 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.158430 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_8afdb494805e4ba7838632b12912ec36 [id=22fff2da3af34564b794765f53773f81]
I20240506 19:57:01.159474 13245 tablet_bootstrap.cc:492] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.161041 13245 tablet_bootstrap.cc:654] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.161329 14910 raft_consensus.cc:482] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.161474 14910 raft_consensus.cc:504] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.162114 14910 leader_election.cc:288] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.162649 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.163802 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_aa42760da3bc4705bb6912cc7ebbfb74 [id=246f3a1786da4e438d563fa8ead150c2]
I20240506 19:57:01.166597 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9eea3fa543d74708aed9bf6f5e378d89 [id=25076b7bd89a4c3bbb205acaab09690a]
I20240506 19:57:01.169986 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3be44884b95d45b4ae8935641eb54286 [id=256783d83a354331a1561cbb217a58ac]
I20240506 19:57:01.171958 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_5b7eae44ba9746f9b088affbf1a88cdb [id=273f7fc51349495f90765ff0b2ba2c2f]
I20240506 19:57:01.174901 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9277b5f5bbf44d878345b1028267ba8a [id=274ffbb5e513489f85dcc6097c8cc444]
I20240506 19:57:01.175568 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_951df9b3f85e416893c271dc056e57ee [id=27ca3baadf324fbe92ecfe455e90428a]
I20240506 19:57:01.179123 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a47ab384cae5499183603602a149e649 [id=28724ef5aa3b4dc2b399336e8959cf34]
I20240506 19:57:01.182740 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_11ace0d42c9840d19108e3bfa5f1bb71 [id=29d018e8e77c4cbfa4cec78985b4de98]
I20240506 19:57:01.183599 13245 tablet_bootstrap.cc:492] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.183846 13245 ts_tablet_manager.cc:1367] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.006s
I20240506 19:57:01.184672 13245 raft_consensus.cc:348] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.184969 13245 raft_consensus.cc:374] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.185072 13245 raft_consensus.cc:729] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.185395 13245 consensus_queue.cc:260] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.186038 13245 ts_tablet_manager.cc:1398] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:01.186370 13245 tablet_bootstrap.cc:492] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.186810 13244 tablet_bootstrap.cc:492] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.187034 13244 ts_tablet_manager.cc:1367] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.002s
I20240506 19:57:01.188177 13245 tablet_bootstrap.cc:654] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.188403 13244 raft_consensus.cc:348] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.188694 13244 raft_consensus.cc:374] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.188817 13244 raft_consensus.cc:729] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.189232 13244 consensus_queue.cc:260] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.190706 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_131e6a1d2bc240439badeffed52cd489 [id=2a02c78fdbd1463987715da7a840f0ce]
I20240506 19:57:01.191677 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b949c48806d45c3a74efbc8503c0bdb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.191988 13120 raft_consensus.cc:2454] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:01.195927 13244 ts_tablet_manager.cc:1398] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:57:01.199517 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97]
I20240506 19:57:01.199990 12922 leader_election.cc:302] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.200525 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_df143abf77c545eab94fa6ed375d125f [id=2c03449e752d48ceaa0e3d59d0d49dd9]
I20240506 19:57:01.200333 14700 raft_consensus.cc:482] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.201932 14700 raft_consensus.cc:504] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.202913 14700 leader_election.cc:288] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.203771 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164]
I20240506 19:57:01.204689 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb9c05d703f417780e77359e19e542e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.205332 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c1b76d968cf047ab833ec4ec4af83697 [id=2c499becf4f44f84bc6d33b9d92d8791]
I20240506 19:57:01.206704 14700 raft_consensus.cc:2781] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.206847 14700 raft_consensus.cc:482] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.207021 14700 raft_consensus.cc:3037] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:01.205595 12921 leader_election.cc:341] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.207903 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_54845e03ef4e4cc7b472aeee0c2b4c42 [id=2e805268d6aa44a681a952f654d3055a]
I20240506 19:57:01.208222 13244 tablet_bootstrap.cc:492] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.211103 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_95c37af6ef334405b443163c4eb60668 [id=2eebc48dfe464f9ba15aec649e8f9d08]
I20240506 19:57:01.211326 14915 raft_consensus.cc:482] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.211452 14915 raft_consensus.cc:504] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.212062 14915 leader_election.cc:288] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.211172 13245 tablet_bootstrap.cc:492] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.212630 13245 ts_tablet_manager.cc:1367] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240506 19:57:01.213057 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.213477 13245 raft_consensus.cc:348] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.213730 13245 raft_consensus.cc:374] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:57:01.213761 13058 leader_election.cc:341] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.213804 13245 raft_consensus.cc:729] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.215274 13245 consensus_queue.cc:260] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.214964 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_960b875a518745a38b1a1e6699ec3a2f [id=2efd55b739244f81a16c345e3d8746af]
I20240506 19:57:01.216704 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_829c185590b0466f86672ee2ce1d349f [id=2f5efbdf6ae84721b52f3e9396ea42f9]
I20240506 19:57:01.217411 13245 ts_tablet_manager.cc:1398] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:01.218178 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_15b63528dc5649619d25b71595773037 [id=30577c164a974b99a479276640cca87e]
I20240506 19:57:01.218869 13245 tablet_bootstrap.cc:492] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.219381 13244 tablet_bootstrap.cc:654] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.220855 13245 tablet_bootstrap.cc:654] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.222530 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3380422dfa66489eb9f22c0b76154c8c [id=31bc34780203473fbb7aa8f32487c8fa]
I20240506 19:57:01.223050 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e [id=330c88093b4b4bcfa831c77ce9b44115]
I20240506 19:57:01.225124 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9b101aa14e0a4b1cbb4aefcc7dc75d2d [id=33729edfc83a4cb19d3594b2ce4c606d]
I20240506 19:57:01.227542 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0f8f0a413eab460aa84362ac194994e1 [id=33dfa30213944d2883cefa6f0c6f9365]
I20240506 19:57:01.230062 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d0df8358053749a49c25b73041023ac1 [id=351289efb08e425da033ce80530da032]
I20240506 19:57:01.232836 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c2d2254e8afc4f8a80b105ceffab0b47 [id=3614ed5dd41b467aa26de18b1b04ed08]
I20240506 19:57:01.234759 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b820628699774953a9a7b655edefe41c [id=362f73af54c04bbc9de5657a30242626]
I20240506 19:57:01.236874 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c7f65b0de80a4c4cbd8a85f397e7a344 [id=36c18887a18442ab9f91deb76c49af64]
I20240506 19:57:01.239418 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0deb4b5d182d46de8bece88de704f3ea [id=3731ecdeeed84de1b6d350e8283966a0]
I20240506 19:57:01.241271 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e]
I20240506 19:57:01.245424 13244 tablet_bootstrap.cc:492] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.245712 13244 ts_tablet_manager.cc:1367] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.004s
I20240506 19:57:01.213270 14700 raft_consensus.cc:504] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.246557 13244 raft_consensus.cc:348] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.246766 13244 raft_consensus.cc:374] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.246826 13244 raft_consensus.cc:729] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.247071 13244 consensus_queue.cc:260] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.247560 14700 leader_election.cc:288] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.248023 13244 ts_tablet_manager.cc:1398] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:01.248903 13244 tablet_bootstrap.cc:492] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.250577 13244 tablet_bootstrap.cc:654] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.251914 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1dabc02b391c4c3386ec45beabfc24ca [id=37f5cb0fe4b348e4a1e0a9d4cec538be]
I20240506 19:57:01.254635 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3bd43b5745da4ad1ab82dceeec1f39b6 [id=38092479ce4249f5aed0ab679d4374ac]
I20240506 19:57:01.258136 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9e229895c2804c09bd30a74901fd77b6 [id=38db8971145547068978042310ba1d0f]
I20240506 19:57:01.261780 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e300dc01d2fc4eb1bb7c22f60f49f570 [id=38df61b9f4a641eeb68e44b1eadf7d1a]
I20240506 19:57:01.264333 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7a1c1e363dbc41dc9708971c2656dfe5 [id=3a6f58959205421cb5894d9424959ea0]
I20240506 19:57:01.267664 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e1bc856560494ef0ac2a3d693b45a7c0 [id=3af64f2f2b324a43add1c324664a45c1]
I20240506 19:57:01.271595 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b2e9c53dcdb14928b637180cdca5f786 [id=3b6c71a27559456aa2d11f0736cf5dbf]
I20240506 19:57:01.273087 13244 tablet_bootstrap.cc:492] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.273363 13244 ts_tablet_manager.cc:1367] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240506 19:57:01.273653 14910 raft_consensus.cc:482] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.273795 14910 raft_consensus.cc:504] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.274209 13244 raft_consensus.cc:348] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.274379 13244 raft_consensus.cc:374] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.274427 13244 raft_consensus.cc:729] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.274458 14910 leader_election.cc:288] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.274645 13244 consensus_queue.cc:260] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.275195 13244 ts_tablet_manager.cc:1398] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:01.275496 13244 tablet_bootstrap.cc:492] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:01.275931 12783 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d43da1d12ce647dcbcd31263932b2f54
I20240506 19:57:01.277065 13244 tablet_bootstrap.cc:654] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.280289 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b24141a2526d4993b9210b8cc83dd50c [id=3bcbaf5c0f19445789fc1a1b3dacd629]
I20240506 19:57:01.280795 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fcad4ec094e649709cb1cf84577e7483 [id=3bfc6a7cd8d940b298a8bd52f65e78d7]
I20240506 19:57:01.281538 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_11d89ae7590b466eacd8a986b0d180a4 [id=3cd8fd373b3e40caa343e05139cacd74]
I20240506 19:57:01.282011 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f26ab51c669247fb9d3624a8c485509b [id=3dcb08c9175142c7831f2224f317a453]
I20240506 19:57:01.288928 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a8f8c4d749444684a5b327c80c5f5829 [id=3e6deafd817f43c0b137bae39fb1d39c]
I20240506 19:57:01.292086 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_bbde4cb2e81b4723a42d8d0c2006c1a0 [id=3ea3309b8a554d64ab44ae10da633751]
I20240506 19:57:01.295760 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9caff027d79b4a8cbdde4ba6074c1e90 [id=3f77354fa69f4941abc4db4fd5b70b7a]
I20240506 19:57:01.296399 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_8ef666d2b87f4076a267d567a422a9c8 [id=4013fcd5ff0545f28ba8f7f5cfc64345]
I20240506 19:57:01.297510 13244 tablet_bootstrap.cc:492] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.297668 13244 ts_tablet_manager.cc:1367] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20240506 19:57:01.298475 13244 raft_consensus.cc:348] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.298681 13244 raft_consensus.cc:374] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.298736 13244 raft_consensus.cc:729] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.298954 13244 consensus_queue.cc:260] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.299757 13244 ts_tablet_manager.cc:1398] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:01.300230 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a1625cd2cd914b31b16e4b86b2aa545d [id=405ec704b2af43628e19824582b94eaa]
I20240506 19:57:01.300469 13244 tablet_bootstrap.cc:492] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.301846 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3dbe1bd9a46f4c9b8146d1be89e93c9d [id=40d6e63d54e44c089dff7b3115e15cdf]
I20240506 19:57:01.302821 13244 tablet_bootstrap.cc:654] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.304280 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ef9202d673db4585a988cbb893c1eed0 [id=42d9312c57544c3cad708124ec023b00]
I20240506 19:57:01.305802 15009 raft_consensus.cc:482] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.306002 15009 raft_consensus.cc:504] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.306138 14910 raft_consensus.cc:482] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.306293 14910 raft_consensus.cc:504] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.306813 15009 leader_election.cc:288] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.306957 14910 leader_election.cc:288] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.307428 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6d330fca725b4de09fa2e8a21d67d676 [id=436998b2930242f8ad542a3a8ae6a91c]
I20240506 19:57:01.307997 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a7a0fc1296414d58a1cc512c170d3e96 [id=436b5c4427c24f4caf61a5c06a923b59]
I20240506 19:57:01.307942 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da091c44824bd6bb8d390a5fb01ef8" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:01.309549 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 109638us
I20240506 19:57:01.310174 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_28ea2a7143b84c1c9eeb79da8bf201ad [id=44207be9fc564b5bba7a4513e0b3c1bd]
I20240506 19:57:01.310369 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.310628 12982 raft_consensus.cc:2454] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:01.311125 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb9c05d703f417780e77359e19e542e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.311317 12987 raft_consensus.cc:2454] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:01.316892 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0316621fc64c4470a110b2ef8a2540ab [id=4448b2536e3649c78cd6f6334852a8e6]
I20240506 19:57:01.317749 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.318104 12986 raft_consensus.cc:3037] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.318670 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d0265b889b7d4b35a1e6df3bcdfbe208 [id=4537e3ced96b436a9e4d134427351495]
I20240506 19:57:01.321771 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b5b09cfa481b48eba1086b706fb28557 [id=4637f24d5dd64c3ea6e62357d9a4e686]
I20240506 19:57:01.322057 13249 tablet_bootstrap.cc:492] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:01.322360 13249 ts_tablet_manager.cc:1367] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.317s user 0.005s sys 0.000s
I20240506 19:57:01.323693 13249 raft_consensus.cc:348] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.323951 13249 raft_consensus.cc:374] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.324024 13249 raft_consensus.cc:729] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.324251 13249 consensus_queue.cc:260] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.324976 13249 ts_tablet_manager.cc:1398] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:57:01.325321 13249 tablet_bootstrap.cc:492] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:01.326368 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7c64e545a04644e798e5a0df2ca79723 [id=469bb2f1eb87405aab9296077b9130ac]
I20240506 19:57:01.327477 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_583d4d82aebd4faaa5506048dbb7b71e [id=47243ad822944b879acaef4663ee6dff]
I20240506 19:57:01.328578 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d9e7986164ffd916d2d842dc61f6e" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.329128 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c659d848fa34442fb605f1757571b5e7" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.329583 14910 raft_consensus.cc:482] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.329736 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43da1d12ce647dcbcd31263932b2f54" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.329737 14910 raft_consensus.cc:504] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.329893 12982 raft_consensus.cc:2454] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:57:01.330057 13244 tablet_bootstrap.cc:492] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.330199 13244 ts_tablet_manager.cc:1367] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.005s
I20240506 19:57:01.330437 14910 leader_election.cc:288] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.330952 13244 raft_consensus.cc:348] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.331127 13244 raft_consensus.cc:374] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.331190 13244 raft_consensus.cc:729] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.331344 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260d7ab458794df496012cb80466f59c" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.331374 13244 consensus_queue.cc:260] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.331885 13244 ts_tablet_manager.cc:1398] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:01.332062 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_91e0d90fa1a442a2a442d06baf1ef73f [id=49164dc63a4a4b00b435277562ee5224]
I20240506 19:57:01.339175 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b9e1af536a204e51a898c9821976d7ac [id=4a489bb59b9545e991edc010e270af80]
I20240506 19:57:01.339820 13244 tablet_bootstrap.cc:492] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.341107 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da091c44824bd6bb8d390a5fb01ef8" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.342615 13244 tablet_bootstrap.cc:654] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.343631 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb9c05d703f417780e77359e19e542e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.343878 13120 raft_consensus.cc:2454] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:01.344159 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b949c48806d45c3a74efbc8503c0bdb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:01.344313 13114 raft_consensus.cc:3037] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.345002 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b0ca455fa29e479f80da85793bcc6414 [id=4a60097281c44b629e3b164245c6b810]
I20240506 19:57:01.347081 13114 raft_consensus.cc:2454] T 2b949c48806d45c3a74efbc8503c0bdb P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:01.348502 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b86c2104b98d4723bfde3500950e640b [id=4d658c76e1e842588a67621ff1d945a0]
I20240506 19:57:01.349540 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b949c48806d45c3a74efbc8503c0bdb" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.352128 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0821a294cda34e98b1de703d3b5f8c1a [id=4d7b052cb45e4c1e9b33fdb0648f12d8]
I20240506 19:57:01.353271 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4ad903e75547fa829b82cc722fef1e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.353806 12783 consensus_peers.cc:485] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31c440555cfe4195b3ae86e1f1eb6a34. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.355281 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_aa81775ed7b04f91a8a7b6a3619fed4c [id=4f6c511cf5d04a3dba1b46ae7177a533]
I20240506 19:57:01.355604 12843 raft_consensus.cc:3037] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.359738 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_092635dc2dec452490c7710c0cc1a37e [id=4fab85c0d2a44f7d9e65f2f9be3b85f9]
I20240506 19:57:01.360970 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ddc8aae93ec243fd9f2aa7841c96bbca [id=505e004369be45e3950f15e43d47a14e]
I20240506 19:57:01.363807 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0f8c273dcd61423a9055682950543663 [id=509b7c79a486427f9003363c784a5cb1]
I20240506 19:57:01.366554 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_68bc366f32d440a9a0ddab8c2bcfb05e [id=52a4f1f2a0b84809a60715f6e5698b90]
I20240506 19:57:01.367035 13244 tablet_bootstrap.cc:492] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.367233 13244 ts_tablet_manager.cc:1367] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.009s
W20240506 19:57:01.367888 12921 leader_election.cc:341] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.368031 13244 raft_consensus.cc:348] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:01.368203 12921 leader_election.cc:341] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.368250 13244 raft_consensus.cc:374] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.368311 13244 raft_consensus.cc:729] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.368531 13244 consensus_queue.cc:260] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.369040 13244 ts_tablet_manager.cc:1398] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:01.370085 13244 tablet_bootstrap.cc:492] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.370599 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_22d4c2a939a34f7c9a0bbd4eb808d058 [id=531e45af97aa43639c67007d553b70c9]
I20240506 19:57:01.371548 14910 raft_consensus.cc:482] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.371676 14910 raft_consensus.cc:504] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.372252 14910 leader_election.cc:288] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.372416 13244 tablet_bootstrap.cc:654] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:01.373770 12783 leader_election.cc:341] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1d53f3daaa2949a2b6883435f031dc37
I20240506 19:57:01.374091 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_204da63924eb46ad9044cbc009296da3 [id=537838979f76429da504e67aa7485fcb]
I20240506 19:57:01.375845 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741]
I20240506 19:57:01.376778 14910 raft_consensus.cc:482] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.376948 14910 raft_consensus.cc:504] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.377759 14910 leader_election.cc:288] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.378350 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_af258c3a0d0a466ea0d34bbfae91eb67 [id=54ebe9481ac5420b8adc3c32a9af518a]
W20240506 19:57:01.378929 12783 leader_election.cc:341] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: c1ff7a3638004012b08c973192c055a5
I20240506 19:57:01.380026 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_de7835c60b14419892d8827447c27f2e [id=561add2e304a497da06ce1abc6054980]
I20240506 19:57:01.380584 14904 raft_consensus.cc:482] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.380728 14904 raft_consensus.cc:504] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.381412 14904 leader_election.cc:288] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.382149 12848 raft_consensus.cc:3037] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.382743 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7511af3d69ea4e9d921a500168622d24 [id=56e200eedf3e4fdfa122be9f5cc65bfd]
I20240506 19:57:01.382977 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.383705 13058 leader_election.cc:341] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.384166 14910 raft_consensus.cc:482] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.384321 14910 raft_consensus.cc:504] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.385076 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_36f124ac875b412fbe30315246e7c43c [id=5844ade1fcc94f6b89fe5bf3ac829779]
I20240506 19:57:01.385416 14910 leader_election.cc:288] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.385828 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9db7aa6cc91143a9be731a84793c16dc [id=58ca5d5422004bcd9950f0b0703b7695]
W20240506 19:57:01.386070 12783 leader_election.cc:341] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 50ef5a2799b447f992207452dbe39645
I20240506 19:57:01.392074 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_36502249055f419692c8fe55e7f7d2ea [id=59360d532f304a3bbab1c482d0ba8c03]
I20240506 19:57:01.393106 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_df06dbccf0ce480489e9ff86ef1be62d [id=59d2236266164cda96142253cc0a869a]
I20240506 19:57:01.395319 13245 tablet_bootstrap.cc:492] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.395464 13245 ts_tablet_manager.cc:1367] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.177s user 0.023s sys 0.013s
I20240506 19:57:01.395727 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c0a9791431af47eaafcd42d67421b82e [id=5a2a63588e864676bbbaf4ab586a819d]
I20240506 19:57:01.396184 13245 raft_consensus.cc:348] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:01.396389 13245 raft_consensus.cc:374] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.396443 13245 raft_consensus.cc:729] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.396632 13245 consensus_queue.cc:260] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:01.396834 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_74372670a4e94691bb3c743dd7da48f8 [id=5aeac8d5f4484b8b906f0e688c1fc012]
I20240506 19:57:01.397133 13245 ts_tablet_manager.cc:1398] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:01.398815 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_767d7448a8524f9c9a96fff950481033 [id=5bf872e62221484ca77c5d51ae682edc]
I20240506 19:57:01.399221 13244 tablet_bootstrap.cc:492] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.399370 13244 ts_tablet_manager.cc:1367] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.007s
I20240506 19:57:01.402107 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7fa5f678736244999b82beddbbb9ddb5 [id=5c1e34eaaf154ca286d558bc58aef267]
I20240506 19:57:01.402878 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_78979d94cf774d05a800befc0a5b9745 [id=5c6305fd4cde4b1a87ae85c9267d759e]
I20240506 19:57:01.404435 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fd50aaa5e06941b0b3fbb1d2e4966cbb [id=5d6532bd90404984afd9c8d0aa099990]
I20240506 19:57:01.406064 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e63955b8a5324e6688d4a2126f9a2315 [id=5eae429c4d3c4a199359ea1c6532b6d6]
W20240506 19:57:01.407245 13058 consensus_peers.cc:485] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:01.407822 13058 consensus_peers.cc:485] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.407899 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ecc91044eebb4ad6b2799b1569836c8c [id=5f96354a891c42e6aed78d7cbb5d6f00]
I20240506 19:57:01.409126 12845 raft_consensus.cc:3037] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.409178 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7604b11105d14a889c9b95c21133e402 [id=617fd0a3b26f40abaee4b419ead18cba]
I20240506 19:57:01.409888 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eea6723475c7480f86f5246eb849bf13 [id=62405a21681d4effaea8f463eb15bf4d]
I20240506 19:57:01.410626 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_8c2dd5f0bc85416b9c98a687c245f5b5 [id=62473581d9f442d993eb190d59c90efe]
I20240506 19:57:01.411280 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f13896e2670e40119184cffaa84b47d9 [id=6253662e32044bea9e43124b223c3b4c]
I20240506 19:57:01.411919 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0f727575df874202b2cdd6ca62d5727f [id=6299eacabfd0428a81fcd18ce665cec0]
I20240506 19:57:01.412557 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_90043eda47664c8fa37013b6730e5fb0 [id=62a40b2784264beb924df20c9798bc62]
I20240506 19:57:01.413528 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_31d45289fb564acaa4d5f2e45801473d [id=63774e46bd574bc6b5e0ad87a84c1a86]
I20240506 19:57:01.413828 13057 leader_election.cc:302] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.413990 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4ad903e75547fa829b82cc722fef1e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.414098 14915 raft_consensus.cc:2781] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.414175 14915 raft_consensus.cc:482] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.414254 14915 raft_consensus.cc:3037] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.414177 13114 raft_consensus.cc:2454] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:01.414425 13057 leader_election.cc:302] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.414821 14904 raft_consensus.cc:2781] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.414901 14904 raft_consensus.cc:482] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.414975 14904 raft_consensus.cc:3037] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.415110 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d2889007e246431bb796c67ca72db0aa [id=63af27bdf8c84d478759499c8437682c]
I20240506 19:57:01.415709 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_728d3e1c6e2f499fbbd7bc0994b341a3 [id=63fb5dfe8fe64ae199a4861c0a175152]
I20240506 19:57:01.416301 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6cc42687d6c545068a39c096f2932dcd [id=64268ea18e3b4fbb9037218bcca5bbe1]
I20240506 19:57:01.416321 14915 raft_consensus.cc:504] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.416888 14915 leader_election.cc:288] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.417223 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b09bd3dfa767452bbaf2a6be079691ce [id=64b49549c03648dab3e6b25e726405bd]
I20240506 19:57:01.417388 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.417704 14904 raft_consensus.cc:504] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:01.417927 13058 leader_election.cc:341] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.417979 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_550853d348524a21be958c7e260c141b [id=64fa4dbbafd6488bbca91ed2c037a762]
I20240506 19:57:01.418303 14904 leader_election.cc:288] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.418607 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a40f3b8780d740ee995e313e40e3b6fa [id=6557800ff36d45959d95996a473ff371]
I20240506 19:57:01.420069 13245 tablet_bootstrap.cc:492] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.420385 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_223163dff40648e9824d80b4ca3a8894 [id=65fbfa6971874ba9a9f388961010594e]
I20240506 19:57:01.418643 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb9c05d703f417780e77359e19e542e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.421041 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1458eb9e9352452f88190029f7dfc864 [id=667533b391c748afb927260245b13130]
W20240506 19:57:01.421283 13058 leader_election.cc:341] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.421739 13245 tablet_bootstrap.cc:654] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.422128 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3482045609cb42209a8222185144e10f [id=66cd489407394793a3016a8b41c2e62a]
I20240506 19:57:01.422852 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_be02490c0e0c43c29452e03dd1d15be0 [id=66efec0f95ae4e269ef46d7c9ddbe0db]
I20240506 19:57:01.424026 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fb5492528b144beeaec025a770bc00f0 [id=672c95de014c4388a14a6605c25de148]
I20240506 19:57:01.424204 12986 raft_consensus.cc:2454] T b6fa6add23b8403ba5a0a65ac6752a5b P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:01.424831 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4da2e480af98486fa1887da89cc86e38 [id=67e4084ed90e4c68b32bbce4ce7d2fe3]
I20240506 19:57:01.425593 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ad11ec97e2384012981d0daae3a8adbd [id=6921d75de60f46e087367fd1b28ce12d]
I20240506 19:57:01.426170 13249 tablet_bootstrap.cc:654] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.426476 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_21bad439d300445a8251e13b351a3b45 [id=69686d06e616442f9e7c5299ca9fc543]
I20240506 19:57:01.427608 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4607cd1fdeea4e3fa1d76a329c0b0191 [id=6a0f034b7a6c409cabb305e961ab987b]
I20240506 19:57:01.428663 14915 consensus_queue.cc:1035] T 0f28c1f55fd14aca86cb00e43b3a6b76 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.428773 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e66674b291e942a4b42aeb308799018f [id=6a7a686c1b3b441c998a25179c38e9b0]
I20240506 19:57:01.429538 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0e783c507b7342758e013828bcca3d66 [id=6aa9f4187bbf422dbff68fefa9a4d13d]
I20240506 19:57:01.430260 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e9acd0a6178643ba98580f34636cd25b [id=6ae71958a02d48b384248581d7d91404]
I20240506 19:57:01.430982 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_300911c21c1140aca1cd2170583017fb [id=6aecb90966cd41f9aa1066cd32246615]
I20240506 19:57:01.431658 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_24285783bcc5438e9af42796d83db179 [id=6b539c4e5b2444daa0f9ae05ed98ff54]
I20240506 19:57:01.434916 13057 leader_election.cc:302] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.435367 14904 raft_consensus.cc:2781] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.435487 14904 raft_consensus.cc:686] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:01.435833 14904 consensus_queue.cc:237] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:01.432287 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2094814385574142ae32d1a198f4b70b [id=6bf661581d744000bd9a200381c26fb8]
I20240506 19:57:01.438861 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2da343363b4b4d9a9d82f1ab83c1c392 [id=6c44cb4662e4478d8325d5d53ba4c363]
I20240506 19:57:01.439678 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_33e9d54058954ad9806125313b789621 [id=6cab7f5bc39a44e187c9098a05c2e4f0]
I20240506 19:57:01.442078 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb9c05d703f417780e77359e19e542e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.442099 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.442178 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.442286 12983 raft_consensus.cc:3037] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.442332 12985 raft_consensus.cc:2454] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:01.442340 12988 raft_consensus.cc:3037] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.443060 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4508525747cc417a8274f7f3e0a71753 [id=6d74597bfc464c69a263031d78500eb8]
I20240506 19:57:01.444394 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d73690e3683b43168f302fd778a0492b [id=6ecedb3d0a914866b8808b76813c1e1a]
I20240506 19:57:01.445148 12983 raft_consensus.cc:2454] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:01.445806 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397]
I20240506 19:57:01.445904 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d53f3daaa2949a2b6883435f031dc37" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.446219 12986 raft_consensus.cc:2454] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:57:01.447609 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eaf5affd39594005b7d132e27a69442a [id=7249e39100454579b86f755187e5aeaa]
I20240506 19:57:01.447988 12988 raft_consensus.cc:2454] T df04fb90230d4bf4a060db993adafd14 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:01.448233 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ef5a2799b447f992207452dbe39645" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.448446 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ff4a6a2744944a95aae0f535c062d338 [id=72698ebd6fc5442099795eeb11cc028b]
I20240506 19:57:01.449095 15015 raft_consensus.cc:482] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.449146 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7c15bd15fd9f442a986827cad7197380 [id=72af3ab851d74256aa8a9a7ad6baaacb]
I20240506 19:57:01.449223 15015 raft_consensus.cc:504] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.449872 15015 leader_election.cc:288] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.450146 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_898e8b5ca87647bf9715e1304ed7f90b [id=73594db848b740e688f6bc386e5fd717]
I20240506 19:57:01.450198 15015 raft_consensus.cc:482] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.450304 15015 raft_consensus.cc:504] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.450789 15015 leader_election.cc:288] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.451617 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_31d441acbe354b93b7056bee4a33a55a [id=746269304afa4d889e5a0eedb55e192b]
I20240506 19:57:01.452674 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4cc2d777e8ad43d49ca9092206e8dc72 [id=74dfbbbb284d48a8bb4fb2626eb67a3b]
I20240506 19:57:01.453477 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c7551611bbef42be87218a270d30383c [id=756ca049e76744888fc42f22467dbb8b]
I20240506 19:57:01.453961 12922 leader_election.cc:302] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.454310 15010 raft_consensus.cc:2741] T 0fb9c05d703f417780e77359e19e542e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240506 19:57:01.454349 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1ed15ea0ea6148908374bd65889713a7 [id=757c6015872c4aa58e3fbd44122e2ad7]
I20240506 19:57:01.455878 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_014acbfce7444396ad12f76368ca683f [id=767292be27b246929ffa0f7244be6ad7]
I20240506 19:57:01.456096 13249 tablet_bootstrap.cc:492] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:01.455979 13244 raft_consensus.cc:348] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.456238 13244 raft_consensus.cc:374] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.456318 13244 raft_consensus.cc:729] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.456463 13249 ts_tablet_manager.cc:1367] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.131s user 0.004s sys 0.000s
I20240506 19:57:01.456544 13244 consensus_queue.cc:260] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.456645 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c30aa0d022c449e8b993bfa4e61fec2e [id=7754f3fa329242a08c91a80f16bebedd]
I20240506 19:57:01.457108 13244 ts_tablet_manager.cc:1398] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20240506 19:57:01.457093 14910 consensus_queue.cc:1035] T 31c440555cfe4195b3ae86e1f1eb6a34 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.457471 13244 tablet_bootstrap.cc:492] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.457504 14910 raft_consensus.cc:482] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.457603 14910 raft_consensus.cc:504] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.457959 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_aff3a2a3bcf74711a29fc36b829be54d [id=77bf1f6b60274489b239a5270fb3d76b]
I20240506 19:57:01.458179 14910 leader_election.cc:288] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.458096 13249 raft_consensus.cc:348] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.458283 13249 raft_consensus.cc:374] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.458348 13249 raft_consensus.cc:729] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.458724 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_cebd347b740c4e249f4ceac983809141 [id=78308bf582904922bfa15bec097a31a6]
I20240506 19:57:01.458932 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ace0ec43ba44060bcec2a50e1f48098" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.459160 13244 tablet_bootstrap.cc:654] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.459650 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_62d8dd9a1336466d8d008ec2ca4ff8a0 [id=788f6eb22d6e41cfbe0fb6144bda15f8]
I20240506 19:57:01.459622 12922 leader_election.cc:302] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.459872 15009 raft_consensus.cc:2781] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.460027 15009 raft_consensus.cc:686] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:01.460347 15009 consensus_queue.cc:237] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.461139 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9bca56a380f341c28cf5c3f683024098 [id=798e81e44a5a46ec962f48302ab404a8]
I20240506 19:57:01.462141 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5 [id=7a0f775e3b564c95bd5b8a1139facf87]
I20240506 19:57:01.464699 13244 tablet_bootstrap.cc:492] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.464727 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c5bb44d113914983a38052bc548f4eac [id=7b1fd733ea494ea5bbe04550506c6f61]
I20240506 19:57:01.464913 13244 ts_tablet_manager.cc:1367] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240506 19:57:01.465680 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_86885529e3ce46e792201516f327c805 [id=7b921f73ecf9462691e34bda425719c8]
I20240506 19:57:01.465778 13244 raft_consensus.cc:348] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.465965 13244 raft_consensus.cc:374] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.466027 13244 raft_consensus.cc:729] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.466233 13244 consensus_queue.cc:260] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.466648 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fb159d58ea054e2883302a7bc6ea228b [id=7bdb2c56b7df4c8f89d4af97bc7b402b]
I20240506 19:57:01.466888 13244 ts_tablet_manager.cc:1398] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:01.467479 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_42f6118c60d04d7aa217e868cb38a562 [id=7ce9557d82c14955b66d3970197b5bee]
I20240506 19:57:01.468238 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9068971de53c4d30bbe309692dd907f3 [id=7d2da7abfa3e48f79147c8a16820ebcb]
I20240506 19:57:01.468979 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7440450ff02b41f78dc29fee0df8b2f0 [id=7d30aa5cd08345d6a0fdca8e6ef092c8]
I20240506 19:57:01.469710 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_641a8c0a4cee47a18707ea343732c1d5 [id=7e017cf459b242b8bce4d98a825de531]
I20240506 19:57:01.470471 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d486818ffe5644efa8b0211637048aa3 [id=7e16a8d752884de9a4b54a8f7a376ddf]
I20240506 19:57:01.471149 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3]
I20240506 19:57:01.471853 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0bcaa070661f4dc28a869fe16f7bc6c0 [id=7e9605522cd24b86adeec813cb0d7697]
I20240506 19:57:01.472314 12922 leader_election.cc:302] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.472604 15010 raft_consensus.cc:2781] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.472690 15010 raft_consensus.cc:482] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.472774 15010 raft_consensus.cc:3037] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.473316 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9ab15c3c5575442181350680a86a5370 [id=7f170fa8d27a423f977115e24d56b038]
I20240506 19:57:01.474371 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_303d9190978b40a89152b877de75abb4 [id=7fe70815a63a40139593dcf8b27b60bf]
I20240506 19:57:01.475369 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9a458e9734bc4cd78dd0eb34dc07783f [id=80227a440b0948cdae9918e92486e003]
I20240506 19:57:01.475664 12781 leader_election.cc:302] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.475813 12846 raft_consensus.cc:3037] T 880ecc75bab446e39c687f3c8b740361 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.475919 15010 raft_consensus.cc:504] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.476101 12781 leader_election.cc:302] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.476490 12781 leader_election.cc:302] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.476540 15010 leader_election.cc:288] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.477376 15031 raft_consensus.cc:2726] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.477536 14910 raft_consensus.cc:2726] T c659d848fa34442fb605f1757571b5e7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.477581 15031 raft_consensus.cc:2781] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.477663 15031 raft_consensus.cc:482] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.477725 15031 raft_consensus.cc:3037] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.478881 13244 tablet_bootstrap.cc:492] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.480928 13244 tablet_bootstrap.cc:654] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.480839 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.481338 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b375490272a4dec9a743f6458f6d6de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.481367 15031 raft_consensus.cc:504] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.482700 12781 leader_election.cc:302] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:57:01.483096 12781 leader_election.cc:341] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 260d7ab458794df496012cb80466f59c
I20240506 19:57:01.483194 12781 leader_election.cc:302] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.483426 14910 raft_consensus.cc:2726] T 45da091c44824bd6bb8d390a5fb01ef8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.483549 14910 raft_consensus.cc:2726] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.483522 14739 consensus_queue.cc:1035] T c8ece0bd3a3b47ef88a0b4910f0efae9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.484864 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_111e3f9425804b4ca0bf599cc9377867 [id=8026f48a5250456a8cea07bb6446b845]
W20240506 19:57:01.486184 12783 leader_election.cc:341] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d43da1d12ce647dcbcd31263932b2f54
I20240506 19:57:01.487002 14797 consensus_queue.cc:1035] T bc85233b180f4b30a86bb65c86af684e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.487514 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_bb460b3bc3c6433aa0d6abe2f3eea2e2 [id=80efe594c8b64a04864ac3a11061edd8]
I20240506 19:57:01.488041 13249 consensus_queue.cc:260] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.489001 13249 ts_tablet_manager.cc:1398] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240506 19:57:01.490785 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4]
I20240506 19:57:01.492462 15031 leader_election.cc:288] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.492703 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_742be09ecdef4a4c9d11965a17109e81 [id=82062552076642299d2d7e511dd6bebf]
I20240506 19:57:01.493431 12848 raft_consensus.cc:3037] T 838d9e7986164ffd916d2d842dc61f6e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:01.494211 12921 consensus_peers.cc:485] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.498884 14797 consensus_queue.cc:1035] T 986f9c83b04f41b49e9b78fce731a90f P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.500449 13249 tablet_bootstrap.cc:492] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:01.501956 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1c043170c0ff4721aeb47b27867693f0 [id=824ad67c294d42f181701cbf63b635c3]
I20240506 19:57:01.502391 13057 leader_election.cc:302] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.502751 14915 raft_consensus.cc:2781] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.502841 14915 raft_consensus.cc:482] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.502910 14915 raft_consensus.cc:3037] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:01.503798 13058 consensus_peers.cc:485] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.503950 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_68b03f1b0b0f468494be25ef614a5b08 [id=825fc426cf02442f9ed682b5af9fda40]
I20240506 19:57:01.504581 13057 leader_election.cc:302] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.504987 14904 raft_consensus.cc:2781] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.505203 14904 raft_consensus.cc:686] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:01.505519 14904 consensus_queue.cc:237] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.505764 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c35eaf4f2426412381b537c1bcb048f7 [id=8439d7bbb943423f9c433fd0c82f1773]
I20240506 19:57:01.506812 14915 raft_consensus.cc:504] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.507757 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b22912910904450c99d5f1fc1be34500 [id=860954ec42cf41e68fad33cb0b6a1e0b]
I20240506 19:57:01.507808 14915 leader_election.cc:288] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.508782 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e2c60bcc57d34618a8cc33cab56b7a78 [id=862ab2f52d2c472eba72311e18532f1a]
I20240506 19:57:01.510515 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1f4f4333e9c147919940536b669bffd3 [id=873dfe33ce7141f2bc25d7575a63fc9f]
I20240506 19:57:01.511207 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_5c27619f39744d3aa10bfc80a3397da5 [id=8879a2545913487dafc8603951a8e8c2]
I20240506 19:57:01.494010 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4ad903e75547fa829b82cc722fef1e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:01.499663 12781 leader_election.cc:341] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: c1ff7a3638004012b08c973192c055a5
I20240506 19:57:01.514005 12781 leader_election.cc:302] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.514650 12781 leader_election.cc:302] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.515043 15023 raft_consensus.cc:2726] T c1ff7a3638004012b08c973192c055a5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.515115 13057 leader_election.cc:302] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.515291 15023 raft_consensus.cc:482] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.515411 15023 raft_consensus.cc:504] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.515614 14915 raft_consensus.cc:2781] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.515889 14915 raft_consensus.cc:686] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:01.516178 14915 consensus_queue.cc:237] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.516683 14910 raft_consensus.cc:2781] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.516770 14910 raft_consensus.cc:482] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.516850 14910 raft_consensus.cc:3037] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.517679 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2fda57476820461185abfb4e54f9a3d9 [id=893f3cabb2d84a349345cc6fbf9f4656]
I20240506 19:57:01.518018 15023 leader_election.cc:288] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.508103 13244 tablet_bootstrap.cc:492] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.508173 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.518738 13244 ts_tablet_manager.cc:1367] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
W20240506 19:57:01.518712 13058 leader_election.cc:341] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:01.518950 12783 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 20aa06290b9a4814a006c423ccd8fc46
I20240506 19:57:01.519430 14910 raft_consensus.cc:504] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.519598 13244 raft_consensus.cc:348] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.519748 13244 raft_consensus.cc:374] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.519807 13244 raft_consensus.cc:729] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.519711 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e1222c6575614a1d83d72a3751290c35 [id=8ac1522e079245629612384cd7cad952]
I20240506 19:57:01.520061 13244 consensus_queue.cc:260] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.520087 14910 leader_election.cc:288] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.520511 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758]
W20240506 19:57:01.521206 12783 leader_election.cc:341] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 1d53f3daaa2949a2b6883435f031dc37
I20240506 19:57:01.521550 13244 ts_tablet_manager.cc:1398] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:57:01.522439 13244 tablet_bootstrap.cc:492] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.524099 13244 tablet_bootstrap.cc:654] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.526641 12781 leader_election.cc:302] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.527096 14910 raft_consensus.cc:2726] T 50ef5a2799b447f992207452dbe39645 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.528416 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b375490272a4dec9a743f6458f6d6de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.528630 13114 raft_consensus.cc:2454] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:01.528546 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:01.530058 13058 consensus_peers.cc:485] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.531221 14915 consensus_queue.cc:1035] T 6089f4f68bfe415e8d97dfd6849b7714 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.531759 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_79eff2000c124513b48aeefee23bb9a3 [id=8b305bf0e98344d1a63fe0c84e5ee0fe]
W20240506 19:57:01.532464 12783 consensus_peers.cc:485] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bece304e0ba048ecbb2d92aa71fdf707. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.536289 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fb84f06ac8d34f659ca376315100395f [id=8b3fcd8705044aba983c11196817167f]
I20240506 19:57:01.539009 13245 tablet_bootstrap.cc:492] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.539157 13245 ts_tablet_manager.cc:1367] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.119s user 0.012s sys 0.016s
I20240506 19:57:01.539927 13245 raft_consensus.cc:348] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.540165 13245 raft_consensus.cc:374] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.540227 13245 raft_consensus.cc:729] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.540498 13245 consensus_queue.cc:260] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.541079 13245 ts_tablet_manager.cc:1398] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:01.542626 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4ad903e75547fa829b82cc722fef1e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:01.542973 13124 raft_consensus.cc:3037] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.545503 14910 raft_consensus.cc:482] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.545660 14910 raft_consensus.cc:504] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.546293 14910 leader_election.cc:288] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.547770 13124 raft_consensus.cc:2454] T 4b4ad903e75547fa829b82cc722fef1e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:57:01.548267 12783 leader_election.cc:341] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 46916e778bf34188959e1cfff084bffc
I20240506 19:57:01.541412 13245 tablet_bootstrap.cc:492] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.549993 14910 raft_consensus.cc:482] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.549921 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b8b3b6c37e2d4fe9a43050bea37c80b7 [id=8e6a813495f84e8b964d966ee6673ade]
I20240506 19:57:01.550215 13245 tablet_bootstrap.cc:654] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.550153 14910 raft_consensus.cc:504] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.550854 14910 leader_election.cc:288] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:01.551839 12783 leader_election.cc:341] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ddae4b0d8d0943d99f54904266f293d4
I20240506 19:57:01.552215 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4a6c1aff66ee42a293a7af16d9c5a002 [id=8e8f6ceb668d4e14a5ad41373ab56a4a]
I20240506 19:57:01.554433 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_096d0d9806624f8dab6c61cfd67e5481 [id=8f3a327c7cff4778ace0d6588fbba7bf]
I20240506 19:57:01.556694 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_84cf0c067fa84ec2b9b35f2f981d4671 [id=8f48c6efec1c40b78353ad7d31c92096]
I20240506 19:57:01.558482 15010 raft_consensus.cc:482] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.558645 15010 raft_consensus.cc:504] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.560338 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_dbebd827d8ed405bae594e5adc6c14a9 [id=9006912cbc764b9a8f90a11d963cf02a]
I20240506 19:57:01.562258 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_680d6530958740e293cd6eddcad88b21 [id=919e8b8b265b452e8972fa41f11645a9]
I20240506 19:57:01.568491 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_acd9eb17729b45d69cd7e0b55fb9e63a [id=959ae33860b1462eb07511168b279534]
I20240506 19:57:01.572000 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_42386049e25d4c309be88fb22d88bc71 [id=979a99794a424140b5069524b475d849]
I20240506 19:57:01.572885 13245 tablet_bootstrap.cc:492] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.573052 13245 ts_tablet_manager.cc:1367] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240506 19:57:01.575428 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772]
I20240506 19:57:01.575951 13245 raft_consensus.cc:348] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.576151 13245 raft_consensus.cc:374] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.576200 13245 raft_consensus.cc:729] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.576382 13245 consensus_queue.cc:260] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.578476 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_23e6be51af4a449a8f17e9b5fed56041 [id=9855dc7eecda4d969b9261f88384d088]
W20240506 19:57:01.579823 12921 leader_election.cc:341] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.582270 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fee8e613a5414e32af0d532017c321b7 [id=99b2a060bcd047588fa6d5e9756ccbe8]
I20240506 19:57:01.582571 15010 leader_election.cc:288] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.582834 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ace0ec43ba44060bcec2a50e1f48098" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:57:01.583341 12921 leader_election.cc:341] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:01.583784 12921 leader_election.cc:341] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.584487 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a24848c14dfc4d0ca604aabf57ec28d4 [id=99b553da739246f1b28a349d9b0f3462]
I20240506 19:57:01.585989 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e04e93a5543f9934077006eaf2ba9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.586194 12844 raft_consensus.cc:2454] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:01.586958 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a7afaab4e68b4077a2876e0a0c12a73b [id=99c895fce27a4cfd97900cf3c10d7633]
I20240506 19:57:01.587064 12842 raft_consensus.cc:3037] T e1b011e94e2b47d7a51d6816576ea124 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.591143 13249 tablet_bootstrap.cc:654] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.591264 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d01da29ce66546ab88b2eba38a69c4d1 [id=9d511b62ea4b4e66a721a3508f15540e]
I20240506 19:57:01.591825 13245 ts_tablet_manager.cc:1398] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240506 19:57:01.592854 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f72793ccf9b146a0baad8e0836ca260c [id=9d6272e3834b49648147d5d86f7ba1a2]
W20240506 19:57:01.594108 12921 consensus_peers.cc:485] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.594373 12921 leader_election.cc:302] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:57:01.594836 15012 raft_consensus.cc:2781] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.594739 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_12084251f64140ce8b5be5e06ed93ed3 [id=9d74bf3b46cf4be1a8bf414b00a003ef]
I20240506 19:57:01.594935 15012 raft_consensus.cc:482] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.595009 15012 raft_consensus.cc:3037] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.595867 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d53f3daaa2949a2b6883435f031dc37" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.596065 12986 raft_consensus.cc:3037] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.596191 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43da1d12ce647dcbcd31263932b2f54" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.596343 12988 raft_consensus.cc:3037] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.597340 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddae4b0d8d0943d99f54904266f293d4" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.598147 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.598302 12983 raft_consensus.cc:3037] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.598260 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_44dc34cdfbc44f1791118c880102c3d5 [id=9f0efd77ee3c48408f0b33c81a2dd76d]
I20240506 19:57:01.599944 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2ba27e19baf9482da781cb704b29bb2b [id=9f92215443d6470f8ff3e7d27cbbf7f0]
I20240506 19:57:01.601558 12781 leader_election.cc:302] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
W20240506 19:57:01.602032 12781 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 20aa06290b9a4814a006c423ccd8fc46
I20240506 19:57:01.602135 12781 leader_election.cc:302] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.602360 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_24269238bd23445a93f9bf17765610af [id=a02df3672ee24494842ca04c4d67e860]
W20240506 19:57:01.602488 12781 leader_election.cc:341] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 46916e778bf34188959e1cfff084bffc
I20240506 19:57:01.602619 12781 leader_election.cc:302] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.603359 15035 raft_consensus.cc:2726] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.603614 15035 raft_consensus.cc:2726] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.603782 15035 raft_consensus.cc:2726] T 46916e778bf34188959e1cfff084bffc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.603942 15035 consensus_queue.cc:1035] T bece304e0ba048ecbb2d92aa71fdf707 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.604331 12986 raft_consensus.cc:2454] T 1d53f3daaa2949a2b6883435f031dc37 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:57:01.606046 13245 tablet_bootstrap.cc:492] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.606583 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_02edd5e110a34ea292b65d41b0963109 [id=a18b40d9fb2a4e348f06bdb0ef822aa8]
I20240506 19:57:01.607303 13244 tablet_bootstrap.cc:492] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.607605 13244 ts_tablet_manager.cc:1367] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.006s
I20240506 19:57:01.607647 12781 leader_election.cc:302] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.608492 13244 raft_consensus.cc:348] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.608682 15035 raft_consensus.cc:2726] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.611698 12983 raft_consensus.cc:2454] T a568db06324a4fd1b58a29ef0831eb71 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:01.611650 15012 raft_consensus.cc:504] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.611824 12988 raft_consensus.cc:2454] T d43da1d12ce647dcbcd31263932b2f54 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:57:01.612298 13244 raft_consensus.cc:374] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.612468 13244 raft_consensus.cc:729] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.612740 15012 leader_election.cc:288] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.612792 13244 consensus_queue.cc:260] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.614809 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb]
W20240506 19:57:01.617890 12921 consensus_peers.cc:485] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.618161 12922 leader_election.cc:302] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.618459 14797 raft_consensus.cc:2781] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.618532 14797 raft_consensus.cc:482] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.618605 14797 raft_consensus.cc:3037] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.618702 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3eff05dacdb8400894bde49d2337d7f3 [id=a332c12a7dca471eb9e79b7974ea26a4]
I20240506 19:57:01.620847 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c39e942c7b1948ce8b2d1e47289bcbd7 [id=a33c46e7c94c466394d4a001735cf479]
I20240506 19:57:01.621502 14797 raft_consensus.cc:504] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.622306 14797 leader_election.cc:288] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.622786 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc]
W20240506 19:57:01.623617 12922 leader_election.cc:341] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.623768 12922 leader_election.cc:302] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.624218 12922 leader_election.cc:302] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.624398 15010 raft_consensus.cc:2726] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:01.625267 15012 raft_consensus.cc:2781] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.627009 14739 raft_consensus.cc:482] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.626850 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_dbd73da3dbb94cc59750f8b4d27a46ee [id=a3a24a54b336458d8e6c2014ab3b38f8]
I20240506 19:57:01.627136 14739 raft_consensus.cc:504] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.627667 14739 leader_election.cc:288] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.629436 15012 raft_consensus.cc:686] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:01.629832 15012 consensus_queue.cc:237] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.630676 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c649504434cb4d69ac0d74733ff3591e [id=a3c712f1c0db496e908d90ae0af131ee]
I20240506 19:57:01.634443 13245 tablet_bootstrap.cc:654] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:01.636564 13058 consensus_peers.cc:485] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.639199 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_bd582985c60a4a99b9f5e608a1d4003a [id=a4f8f81ef4de4f1589dd6102b92e44ac]
I20240506 19:57:01.643877 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e6692a9b138044fcb858cbed4ae939ed [id=a55ec5afcb4b4c2895ff47bb6252c5c4]
W20240506 19:57:01.645071 12922 leader_election.cc:341] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 373e04e93a5543f9934077006eaf2ba9
I20240506 19:57:01.645606 15015 consensus_queue.cc:1035] T 55d9f32618b64ffa92b5a868081c36a6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.653872 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_aa2cfdcd14e74072be6f657fcb05127e [id=a5c3329d55274c708fa24ff378b7cea0]
I20240506 19:57:01.614734 12847 raft_consensus.cc:3037] T 6ace0ec43ba44060bcec2a50e1f48098 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.659147 13244 ts_tablet_manager.cc:1398] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20240506 19:57:01.659535 13244 tablet_bootstrap.cc:492] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.615231 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e04e93a5543f9934077006eaf2ba9" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.623193 12845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b375490272a4dec9a743f6458f6d6de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.662062 12848 raft_consensus.cc:3037] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.629971 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c32eba2031446ab419cac64068068e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.662357 12921 leader_election.cc:341] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:01.662649 12921 leader_election.cc:341] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.664059 12781 leader_election.cc:302] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.664505 15035 raft_consensus.cc:2781] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.664633 15035 raft_consensus.cc:686] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:57:01.664922 15035 consensus_queue.cc:237] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.665441 12848 raft_consensus.cc:2454] T 373e04e93a5543f9934077006eaf2ba9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:01.666392 12921 leader_election.cc:302] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters:
I20240506 19:57:01.666712 14797 raft_consensus.cc:2781] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.666862 14797 raft_consensus.cc:686] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:01.667430 14797 consensus_queue.cc:237] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.667513 14904 consensus_queue.cc:1035] T 64485c3c13e94055a019f2ef7be1f2fe P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.671687 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0b1a8466d287416ab4ec1f38a4687e69 [id=a7fd74ddf9f24e12bdb816bdd9c65fd2]
I20240506 19:57:01.673758 13057 leader_election.cc:302] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.674494 12781 leader_election.cc:302] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:01.675016 14910 raft_consensus.cc:2781] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.675171 14910 raft_consensus.cc:686] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:57:01.675447 14910 consensus_queue.cc:237] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.676343 14656 raft_consensus.cc:2781] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.676976 14656 raft_consensus.cc:686] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:01.677331 14656 consensus_queue.cc:237] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.677485 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b75f9589d33c4ca6b64d52bbf4bf7bca [id=a867a0cd1eca411a85db8f03ee181d2e]
I20240506 19:57:01.683360 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_659a4d19b7d447438d293f59be74f51a [id=a882030e617646b886caac521fb79161]
I20240506 19:57:01.691488 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_23747e8c1b314e4eaa4566b557389261 [id=a8b5f55150a54dcf8a19dc9e1153ea87]
I20240506 19:57:01.698419 14904 raft_consensus.cc:482] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.698655 14904 raft_consensus.cc:504] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.699960 14904 leader_election.cc:288] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.700269 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.700992 13058 leader_election.cc:341] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.703908 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1104e7d94ecc44c2917aced82c7f7baa [id=a9c8c600d82f4fcd81f4b42b67fe2cf5]
I20240506 19:57:01.707361 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b375490272a4dec9a743f6458f6d6de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:01.707594 13124 raft_consensus.cc:3037] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.707825 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c32eba2031446ab419cac64068068e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:01.707983 13114 raft_consensus.cc:2454] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:01.711985 13124 raft_consensus.cc:2454] T 5b375490272a4dec9a743f6458f6d6de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:01.716753 13245 tablet_bootstrap.cc:492] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.718036 13245 ts_tablet_manager.cc:1367] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.112s user 0.008s sys 0.014s
I20240506 19:57:01.719296 13245 raft_consensus.cc:348] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.723872 15063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27d4482cd59a4e5687992b1f8a177373)
I20240506 19:57:01.725733 13245 raft_consensus.cc:374] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.725390 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d59316fe0b304bd9b03da5f95408a1c6 [id=ac0916562c14439e91b1117812475f5d]
I20240506 19:57:01.726354 13245 raft_consensus.cc:729] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.726086 15063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:57:01.729508 13244 tablet_bootstrap.cc:654] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.733194 13245 consensus_queue.cc:260] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.734876 13245 ts_tablet_manager.cc:1398] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240506 19:57:01.735994 13245 tablet_bootstrap.cc:492] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.735460 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9c5d2e15a9e64b618f68377dce21b541 [id=ad10dbcda08d48c79953b88a4de5eaa6]
I20240506 19:57:01.736286 15063 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:57:01.737735 13245 tablet_bootstrap.cc:654] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.741714 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.741956 12982 raft_consensus.cc:2454] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:01.741561 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9f13287a26d24d28813ef3727d8e6844 [id=ad17c57f0f784939a3f5413480094d96]
I20240506 19:57:01.744405 13245 tablet_bootstrap.cc:492] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.744563 13245 ts_tablet_manager.cc:1367] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240506 19:57:01.745540 13245 raft_consensus.cc:348] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.745772 13245 raft_consensus.cc:374] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.745833 13245 raft_consensus.cc:729] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.747916 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_76d78f0c434c45508bfd15806c510293 [id=ada2217dde264b8dabf49f9a8bcb25c4]
I20240506 19:57:01.746063 13245 consensus_queue.cc:260] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.754245 13245 ts_tablet_manager.cc:1398] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240506 19:57:01.762463 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c280dff5527d4a95ad135ceba07f8a7f [id=ae97ea5c98db4ea4baf292ee3eef4136]
I20240506 19:57:01.768890 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1b1ba78c0a194d509cee193f3bec562e [id=af9eea916c5f44b2a0c650481873e4f9]
I20240506 19:57:01.776605 13249 tablet_bootstrap.cc:492] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:01.777439 13245 tablet_bootstrap.cc:492] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.779181 13245 tablet_bootstrap.cc:654] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.781245 13249 ts_tablet_manager.cc:1367] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.292s user 0.012s sys 0.000s
I20240506 19:57:01.782431 13249 raft_consensus.cc:348] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.783179 14915 consensus_queue.cc:1035] T 9fed2f12eda54be0879b1e9c33ca3e31 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.784427 14915 consensus_queue.cc:1035] T 76e6f7555dfa4122ac00bae7b9e733fe P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.783767 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a881b5a2186a46389e703fdf3f053656 [id=b067ce55300045479fc95e5bbdb86589]
I20240506 19:57:01.785209 13057 leader_election.cc:302] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.785665 14915 raft_consensus.cc:2781] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.785742 14915 raft_consensus.cc:482] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.785800 14915 raft_consensus.cc:3037] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.788866 14915 raft_consensus.cc:504] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.790056 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_15973904471f431bb1fa23a0e060548a [id=b06afb6718ed4fbe8065e5fa6d3bfcbe]
I20240506 19:57:01.790445 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:01.791042 13058 leader_election.cc:341] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.791244 14915 leader_election.cc:288] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:01.795539 14089 leader_election.cc:334] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111)
I20240506 19:57:01.796958 13249 raft_consensus.cc:374] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.797196 13249 raft_consensus.cc:729] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:57:01.790633 12922 leader_election.cc:341] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 373e04e93a5543f9934077006eaf2ba9
I20240506 19:57:01.798266 14797 consensus_queue.cc:1035] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.798688 12922 leader_election.cc:302] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.798929 14797 raft_consensus.cc:2781] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:01.799055 14797 raft_consensus.cc:482] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:01.799154 14797 raft_consensus.cc:3037] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.799477 12922 leader_election.cc:302] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.799716 14700 raft_consensus.cc:2781] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.799829 14700 raft_consensus.cc:686] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:01.800133 14700 consensus_queue.cc:237] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.804231 13249 consensus_queue.cc:260] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.804165 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_629374150cbb4998973db994df329a42 [id=b12b185a8f664dada902a4448a7a29aa]
I20240506 19:57:01.809773 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_456c37fe3dda48289f407577c6618ba9 [id=b164ee48c45e4826868242c03700533e]
I20240506 19:57:01.810446 13245 tablet_bootstrap.cc:492] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.810675 13245 ts_tablet_manager.cc:1367] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
I20240506 19:57:01.810616 13244 tablet_bootstrap.cc:492] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.811313 13249 ts_tablet_manager.cc:1398] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240506 19:57:01.811817 13245 raft_consensus.cc:348] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.812076 13245 raft_consensus.cc:374] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.812179 13245 raft_consensus.cc:729] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.812506 13245 consensus_queue.cc:260] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.811944 13244 ts_tablet_manager.cc:1367] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.152s user 0.001s sys 0.003s
I20240506 19:57:01.802212 14797 raft_consensus.cc:504] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.813997 14797 leader_election.cc:288] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.814990 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c32eba2031446ab419cac64068068e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:01.815153 13245 ts_tablet_manager.cc:1398] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:57:01.815209 13249 tablet_bootstrap.cc:492] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:57:01.815671 12921 leader_election.cc:341] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.818852 13245 tablet_bootstrap.cc:492] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.819466 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1590f38624f04f70a068b387db86e4bc [id=b21b01c7b4d146c5874a19d6621fb369]
I20240506 19:57:01.821269 13245 tablet_bootstrap.cc:654] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.822058 13244 raft_consensus.cc:348] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.824313 13244 raft_consensus.cc:374] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.825452 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:01.825632 12988 raft_consensus.cc:3037] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.824921 13244 raft_consensus.cc:729] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.826861 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c32eba2031446ab419cac64068068e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:01.827040 13116 raft_consensus.cc:3037] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.828176 14949 consensus_queue.cc:1035] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:01.830861 13116 raft_consensus.cc:2454] T 01c32eba2031446ab419cac64068068e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:01.828392 13244 consensus_queue.cc:260] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:01.835232 12921 consensus_peers.cc:485] T de26793c006440dc9aecc07963ddb85e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:01.837630 12847 raft_consensus.cc:3037] T ddae4b0d8d0943d99f54904266f293d4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.843410 14904 raft_consensus.cc:482] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.843024 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_92a91e3ad2a7411b8cdf06c227b34759 [id=b38b4a611e264dc7b6dfff5d561692e0]
I20240506 19:57:01.843566 14904 raft_consensus.cc:504] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.843887 13244 ts_tablet_manager.cc:1398] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.031s user 0.002s sys 0.002s
I20240506 19:57:01.844384 14904 leader_election.cc:288] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:01.846894 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b89fb24d94042e1bfa772265a991680" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:01.847543 12842 raft_consensus.cc:3037] T 260d7ab458794df496012cb80466f59c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:01.851883 13058 consensus_peers.cc:485] T 028055b9df1f4667bd2e54909690e327 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:01.853668 13058 leader_election.cc:341] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.861389 13244 tablet_bootstrap.cc:492] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.863678 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_71d092610e224787956867c83b11d4d7 [id=b39d38ddf7f4433fb3fb1ed1c6400a5c]
I20240506 19:57:01.866250 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7e80a457061942da9dbca15a6b063d8e [id=b482961e445b4643a9b9e33fee0cc36b]
I20240506 19:57:01.872939 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_10658c410e1648df879bdf21d2293b2e [id=b4856100088a41d083991b82b8185f1c]
I20240506 19:57:01.881722 13244 tablet_bootstrap.cc:654] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.883570 13245 tablet_bootstrap.cc:492] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.884131 12922 leader_election.cc:302] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.884425 13245 ts_tablet_manager.cc:1367] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.005s
I20240506 19:57:01.885407 14797 raft_consensus.cc:2781] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.885601 14797 raft_consensus.cc:686] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:01.886166 12988 raft_consensus.cc:2454] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:01.886456 14797 consensus_queue.cc:237] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.889968 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b89fb24d94042e1bfa772265a991680" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:01.890156 12987 raft_consensus.cc:2454] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:01.893150 13057 leader_election.cc:302] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:01.893764 14656 raft_consensus.cc:2781] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:01.893885 14656 raft_consensus.cc:686] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:01.894125 14656 consensus_queue.cc:237] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.902667 12848 raft_consensus.cc:3037] T 7b803ae289344f88b604a2053881d59b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:01.903401 13249 tablet_bootstrap.cc:654] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.905436 13245 raft_consensus.cc:348] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.905690 13245 raft_consensus.cc:374] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.906049 13245 raft_consensus.cc:729] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.882845 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_03fba11fcfa745868fa55d4fe134c993 [id=b4cf5ca7aeb145caaea7b343bba253b2]
I20240506 19:57:01.910272 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_5638dbb8bb784639b338a3f8d5a1b492 [id=b5cf40b4c8a948fa94b55af76ae85e8d]
I20240506 19:57:01.911088 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_86e2e3b68d354e02b29118d063456d4a [id=b6aeaf0e19d746dab4a37005dc753d65]
I20240506 19:57:01.910739 13245 consensus_queue.cc:260] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.914033 13245 ts_tablet_manager.cc:1398] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240506 19:57:01.914906 13244 tablet_bootstrap.cc:492] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.915299 13244 ts_tablet_manager.cc:1367] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.001s
I20240506 19:57:01.918422 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02]
I20240506 19:57:01.924401 13245 tablet_bootstrap.cc:492] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:01.927628 13244 raft_consensus.cc:348] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.929601 13244 raft_consensus.cc:374] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.930012 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b35d5d1b38604d24bf72370ec6d5998c [id=b992a884276341348ee0926305500131]
I20240506 19:57:01.930624 13244 raft_consensus.cc:729] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.932447 13244 consensus_queue.cc:260] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.934439 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_085ebc849e674a89a8b03e33327a342d [id=ba90b3aba0c346559318504ce789c81e]
I20240506 19:57:01.937572 13245 tablet_bootstrap.cc:654] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.939106 13244 ts_tablet_manager.cc:1398] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240506 19:57:01.940086 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_686d4b3ab7454f2e87e3fab0797c3cee [id=bae798c3d8fb4998977e22133b0c6d98]
I20240506 19:57:01.941871 13244 tablet_bootstrap.cc:492] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:01.943960 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_5eb85e351b7645f5925a458693f85835 [id=bb93eef6ddeb4f759f3db72b85d798b5]
I20240506 19:57:01.945711 13249 tablet_bootstrap.cc:492] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:01.945904 13249 ts_tablet_manager.cc:1367] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.001s
I20240506 19:57:01.946810 13249 raft_consensus.cc:348] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.947036 13249 raft_consensus.cc:374] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.947124 13249 raft_consensus.cc:729] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.947377 13249 consensus_queue.cc:260] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.948626 13249 ts_tablet_manager.cc:1398] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 19:57:01.948647 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_503b7b916de7483fa1a71e6e6db2cc48 [id=bbba48ab4c994ffc85fd595ebcb98093]
I20240506 19:57:01.949891 13249 tablet_bootstrap.cc:492] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:01.952338 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_2cc2d44f3f26460e8b237e82572aab2b [id=bbc91540b49b4a95ac2d2e0e371c843a]
I20240506 19:57:01.954702 13244 tablet_bootstrap.cc:654] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.956231 13249 tablet_bootstrap.cc:654] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:01.956120 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_65858cfb30f343b5ae7629f09d0883b0 [id=bbe4d5d027284555846908c5b7b5eb88]
I20240506 19:57:01.959592 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eb4ba3a6295e43989a12e36aa7dcc41a [id=bc157254562147ae9e5ffc52a438affa]
I20240506 19:57:01.962055 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1c4853e227ec4b02ab3929603dbba13e [id=bc316585ccd5468cba6c1016998d5d21]
I20240506 19:57:01.963749 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_be0171ec0e714233b40b175da1e82b4d [id=be73b2a1315a45baa1db6feb44a3843b]
I20240506 19:57:01.965618 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0119e18037bb4223ad5b87a035d96386 [id=be8981747042401ebf0f6f583b71d220]
I20240506 19:57:01.967242 14700 raft_consensus.cc:482] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:01.967398 14700 raft_consensus.cc:504] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.968086 14700 leader_election.cc:288] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:01.968745 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08535fa1600b41ff919fefa93cfc4c78" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:01.969347 12921 leader_election.cc:341] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:01.972354 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_be62a1b5f7694bb1b6e32c3e5a843906 [id=be9182dde02a4dab9010593c3ae1ad8b]
I20240506 19:57:01.976471 13244 tablet_bootstrap.cc:492] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:01.976739 13244 ts_tablet_manager.cc:1367] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240506 19:57:01.978433 13244 raft_consensus.cc:348] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.978860 13244 raft_consensus.cc:374] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.979069 13244 raft_consensus.cc:729] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.979367 13249 tablet_bootstrap.cc:492] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:01.979545 13249 ts_tablet_manager.cc:1367] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20240506 19:57:01.979585 13244 consensus_queue.cc:260] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.981225 13249 raft_consensus.cc:348] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.981496 13249 raft_consensus.cc:374] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:01.981552 13249 raft_consensus.cc:729] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:01.981806 13249 consensus_queue.cc:260] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:01.988638 13249 ts_tablet_manager.cc:1398] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240506 19:57:01.991946 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eb33ab4810b94486b70e278e548b02ba [id=bea1ca1264dc4a44a9c2e91d52ae2a81]
I20240506 19:57:01.993427 13244 ts_tablet_manager.cc:1398] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240506 19:57:01.995108 13245 tablet_bootstrap.cc:492] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:01.995371 13245 ts_tablet_manager.cc:1367] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.008s
I20240506 19:57:01.997623 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f09a4c6b266742568b795b84007e3bf0 [id=bedd1d32d0e844608dcb130ef224060e]
I20240506 19:57:01.998652 13249 tablet_bootstrap.cc:492] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:01.998272 13245 raft_consensus.cc:348] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.000126 13245 raft_consensus.cc:374] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.000294 13249 tablet_bootstrap.cc:654] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.000305 13245 raft_consensus.cc:729] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.000669 13244 tablet_bootstrap.cc:492] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.000756 13245 consensus_queue.cc:260] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.008383 13245 ts_tablet_manager.cc:1398] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240506 19:57:02.010620 13245 tablet_bootstrap.cc:492] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.012768 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f05ee8e1debb411c9f6f7be6e92288fd [id=bf509087bead440784afb222450dbcc4]
I20240506 19:57:02.019280 13245 tablet_bootstrap.cc:654] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.020015 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_294ef89cfe854b01a260a32748b84c85 [id=c266c557aaf24c0d9eaba0cc04b6988f]
I20240506 19:57:02.020495 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c]
I20240506 19:57:02.022917 14656 raft_consensus.cc:482] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.023036 14656 raft_consensus.cc:504] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.023600 14656 leader_election.cc:288] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.028947 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d8de6fedf82542b3ae58875e55aca5f2 [id=c3d1889f7f9c43d4bf472af7e20bed70]
I20240506 19:57:02.034515 13057 leader_election.cc:302] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
W20240506 19:57:02.034766 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.194:43527, user_credentials={real_user=slave}} blocked reactor thread for 125142us
I20240506 19:57:02.035624 14656 raft_consensus.cc:2781] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.035722 14656 raft_consensus.cc:482] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.035686 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a396afff3b243ef84cb3dc3836fcdfd" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.035782 14656 raft_consensus.cc:3037] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.038316 14656 raft_consensus.cc:504] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.038486 13244 tablet_bootstrap.cc:654] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.038900 14656 leader_election.cc:288] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.039029 14656 raft_consensus.cc:482] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.039114 14656 raft_consensus.cc:504] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.039541 14656 leader_election.cc:288] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.039664 14656 consensus_queue.cc:1035] T df160fd5d8ce4bc192913ce702f8f478 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.041759 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b5789735f41642769315e305be4cc6ca [id=c4798c42209a4c5eabce8b5053715386]
I20240506 19:57:02.044548 13249 tablet_bootstrap.cc:492] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.044770 13249 ts_tablet_manager.cc:1367] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.007s
I20240506 19:57:02.045697 13249 raft_consensus.cc:348] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.045966 13249 raft_consensus.cc:374] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.046072 13249 raft_consensus.cc:729] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.046363 13249 consensus_queue.cc:260] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.047710 13249 ts_tablet_manager.cc:1398] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:02.047892 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_59e7933756ae440cb5377ffc3ca93c29 [id=c68d44dcff8e4c528981ab7206ee4dcb]
I20240506 19:57:02.048439 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ba47b7bc355a4589a2b439289c6a61de [id=c6e62c9299cf403ba333d67e1dee2dcc]
I20240506 19:57:02.049196 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ab5b8e26274d4c17aafc75dabab5539a [id=c749aef4b9e0450fada39042d70c1917]
I20240506 19:57:02.049571 13249 tablet_bootstrap.cc:492] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.050631 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_1f73021793a44d038dfb378928d261d3 [id=c7ee5ba59ccf47c39ebbce23f5663c3d]
I20240506 19:57:02.051299 13249 tablet_bootstrap.cc:654] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.052297 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_16fd6d89a90c41eaaa2e17854af6470e [id=c91b5a199c00499186aabe5137b4f59c]
I20240506 19:57:02.059688 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d56b1e730d4c4eee9b775d01f57057c5 [id=c96795bdf3654876bd8d7ea4306f7b81]
I20240506 19:57:02.060335 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b879654fd2304197a107fc833fd81e21 [id=c9ddd47c049f435eafb85d7bd033cc2e]
I20240506 19:57:02.061098 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_10189ed452404f5d90cc93ca6b59e7e9 [id=ca65b8a1c8ef4c9b8a04d85fee307005]
I20240506 19:57:02.062309 14949 consensus_queue.cc:1035] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.063071 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_72a13ea44ea64858911b54c9e99739fc [id=cae4b61f8edf46ce85f5f517ced02305]
I20240506 19:57:02.063783 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_badaf4211410493f99e8fd4813c8b489 [id=cb7a5f36e980482faf56e55a16ca3b08]
I20240506 19:57:02.066756 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_8c4e4e8dbeff40b6b23e5456dbb53729 [id=cb8610d949ac4b1f83b8646f3b7d2caf]
I20240506 19:57:02.068516 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_491b0792c74c4700b9c1b789c3ad16d9 [id=cb87e13b10794ab6a279c31169d8afb4]
I20240506 19:57:02.070375 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c05d8864f0ae4794a8a633bd250e7569 [id=cbff41d5d7a8485bb654d68d512fb589]
W20240506 19:57:02.070878 13057 leader_election.cc:341] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.071552 12982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b89fb24d94042e1bfa772265a991680" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:02.071790 12982 raft_consensus.cc:3037] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.075139 12982 raft_consensus.cc:2454] T 1b89fb24d94042e1bfa772265a991680 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:02.075441 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ce8749846847108edf5f554139cbd9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.075717 12985 raft_consensus.cc:2454] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:02.078234 13057 leader_election.cc:302] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
W20240506 19:57:02.078622 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 225067us
I20240506 19:57:02.078743 15104 raft_consensus.cc:2781] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.079025 15104 raft_consensus.cc:686] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:02.079404 12847 raft_consensus.cc:3037] T 59b022c086d141a19a9b35c4a8c8bef9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.079255 15104 consensus_queue.cc:237] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.081954 13057 leader_election.cc:302] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.082755 14915 raft_consensus.cc:2781] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.082845 14915 raft_consensus.cc:482] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.082947 13249 tablet_bootstrap.cc:492] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.083128 13249 ts_tablet_manager.cc:1367] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20240506 19:57:02.082978 14915 raft_consensus.cc:3037] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.086984 15009 raft_consensus.cc:482] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.087148 15009 raft_consensus.cc:504] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.087800 15009 leader_election.cc:288] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.089370 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e073296ce46e786362aaed96eb4dd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:02.090092 12921 leader_election.cc:341] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.093773 13244 tablet_bootstrap.cc:492] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.094421 13244 ts_tablet_manager.cc:1367] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.004s
I20240506 19:57:02.096889 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a396afff3b243ef84cb3dc3836fcdfd" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:02.096980 13249 raft_consensus.cc:348] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.098239 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b89fb24d94042e1bfa772265a991680" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.099380 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ce8749846847108edf5f554139cbd9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:02.099275 14915 raft_consensus.cc:504] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.102056 13249 raft_consensus.cc:374] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.102536 13249 raft_consensus.cc:729] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.102986 13245 tablet_bootstrap.cc:492] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.095708 13244 raft_consensus.cc:348] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.104264 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ce8749846847108edf5f554139cbd9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:02.104451 12985 raft_consensus.cc:3037] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.104492 13244 raft_consensus.cc:374] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.105382 13244 raft_consensus.cc:729] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.103875 13249 consensus_queue.cc:260] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.104074 14915 leader_election.cc:288] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.107791 12985 raft_consensus.cc:2454] T 44ce8749846847108edf5f554139cbd9 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:02.110020 13057 leader_election.cc:302] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.110394 14915 raft_consensus.cc:2781] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.110607 14915 raft_consensus.cc:686] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:02.111096 14915 consensus_queue.cc:237] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.113258 13249 ts_tablet_manager.cc:1398] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.029s user 0.003s sys 0.002s
I20240506 19:57:02.119557 13249 tablet_bootstrap.cc:492] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.129729 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e53451477cc247e6b349f6f3be94dbd0 [id=cc6062d3d1ce4cba97ae9ef1b304dce6]
I20240506 19:57:02.129871 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08535fa1600b41ff919fefa93cfc4c78" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.130105 13120 raft_consensus.cc:2454] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:02.105638 13245 ts_tablet_manager.cc:1367] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.003s
I20240506 19:57:02.133338 13245 raft_consensus.cc:348] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.133574 13245 raft_consensus.cc:374] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.133669 13245 raft_consensus.cc:729] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.133924 13245 consensus_queue.cc:260] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.134588 13245 ts_tablet_manager.cc:1398] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:02.134898 13245 tablet_bootstrap.cc:492] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.136505 13245 tablet_bootstrap.cc:654] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.136895 13244 consensus_queue.cc:260] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.153755 13244 ts_tablet_manager.cc:1398] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.059s user 0.000s sys 0.002s
I20240506 19:57:02.154881 13244 tablet_bootstrap.cc:492] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.157083 13244 tablet_bootstrap.cc:654] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.168035 13249 tablet_bootstrap.cc:654] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.161898 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e073296ce46e786362aaed96eb4dd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.171343 13120 raft_consensus.cc:2454] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:57:02.165549 13058 leader_election.cc:341] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.171690 13058 leader_election.cc:302] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.171983 15103 raft_consensus.cc:2726] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:02.172363 13058 leader_election.cc:341] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:02.172744 13058 leader_election.cc:341] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.180598 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ce8749846847108edf5f554139cbd9" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.186043 13244 tablet_bootstrap.cc:492] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.186443 14797 consensus_queue.cc:1035] T cade6b1c5a034b23aa33b2349e1dab95 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.189827 12922 leader_election.cc:302] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.190320 14797 raft_consensus.cc:2781] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.190446 14797 raft_consensus.cc:482] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.190562 14797 raft_consensus.cc:3037] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.194669 13244 ts_tablet_manager.cc:1367] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.004s
I20240506 19:57:02.202673 14797 raft_consensus.cc:504] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.203568 13244 raft_consensus.cc:348] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.203245 12922 leader_election.cc:302] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.204444 15009 raft_consensus.cc:2781] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.204538 15009 raft_consensus.cc:482] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.204633 15009 raft_consensus.cc:3037] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.208015 15009 raft_consensus.cc:504] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.206136 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c]
I20240506 19:57:02.213364 15009 leader_election.cc:288] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.213454 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_86e7dc4a07f542679409dcc2a3187f46 [id=cdb79dcc2a134fb788f6f1e285ac593d]
I20240506 19:57:02.214159 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_02b297f901384fd19bf275bcda8a2cab [id=ce3814e9575f446c9f0eec5e6b5e6d63]
I20240506 19:57:02.218600 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e073296ce46e786362aaed96eb4dd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:02.221961 12921 leader_election.cc:341] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.222281 15009 raft_consensus.cc:482] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.222451 15009 raft_consensus.cc:504] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.223385 15009 leader_election.cc:288] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.223942 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aae034f52645ba96827923f4fa305c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:02.224514 12921 leader_election.cc:341] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.229676 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ff37f251dcdb4e1d96089c80f2ed4e82 [id=ce68809ebf714446b16aab8bd6f65702]
I20240506 19:57:02.230202 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9a8bd42081bd47fdb4e31ba99d96756e [id=cefd5538663b400aaf74cbe5e2d56673]
I20240506 19:57:02.231663 13244 raft_consensus.cc:374] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.231787 13244 raft_consensus.cc:729] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.232260 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_da8cee75a0c844e1a4956550119cc252 [id=cf1b4de7eeaa4aabaa3dc7c5160ec86d]
I20240506 19:57:02.238287 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_cf12763862094c8197c09b72b43f4616 [id=cf9c889d67414705b6bda662c1bc3e8a]
I20240506 19:57:02.239622 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08535fa1600b41ff919fefa93cfc4c78" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.240199 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_faeb0284bba8473ebdd24f032788bdf7 [id=cfe6ab0aa2c14d0b8b15653edde3dae4]
W20240506 19:57:02.240242 12921 leader_election.cc:341] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.240619 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e073296ce46e786362aaed96eb4dd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:02.240808 13124 raft_consensus.cc:3037] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.243500 13124 raft_consensus.cc:2454] T ea7e073296ce46e786362aaed96eb4dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:02.243574 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_49169eeefc1442ce942d129a96fc7e55 [id=d0387abfe73c4e16a35a7feef5a12146]
I20240506 19:57:02.244035 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_74ef5084696549ca903c62d3a459320b [id=d0a5c899bb444400a13fd2bdcedb8c70]
I20240506 19:57:02.244747 13245 tablet_bootstrap.cc:492] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.244918 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9324755e6fa74275a4fd6a9a86c63ba6 [id=d2cce1c843f04e4e84256b94e60207b3]
I20240506 19:57:02.244959 13245 ts_tablet_manager.cc:1367] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.001s
I20240506 19:57:02.246444 13245 raft_consensus.cc:348] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.246752 13245 raft_consensus.cc:374] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.246831 13245 raft_consensus.cc:729] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.247545 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08535fa1600b41ff919fefa93cfc4c78" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:02.248255 13124 raft_consensus.cc:3037] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.248343 13244 consensus_queue.cc:260] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.245379 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_fd059b81c3d045c48dbba6b7f1f2e31d [id=d2e2ea2858eb4b39b082df599d497112]
I20240506 19:57:02.249080 14797 leader_election.cc:288] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.249572 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aae034f52645ba96827923f4fa305c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.249804 13120 raft_consensus.cc:2454] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:02.250017 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6013611882be454da8ec1ea823261f6a [id=d2ffd5634f2042b7a85cbe74d6ca1df5]
W20240506 19:57:02.247786 13058 leader_election.cc:341] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.250864 13244 ts_tablet_manager.cc:1398] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.053s user 0.001s sys 0.001s
I20240506 19:57:02.251150 13124 raft_consensus.cc:2454] T 08535fa1600b41ff919fefa93cfc4c78 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:02.251183 13249 tablet_bootstrap.cc:492] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.251286 13244 tablet_bootstrap.cc:492] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.251353 13249 ts_tablet_manager.cc:1367] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.000s
I20240506 19:57:02.253031 13244 tablet_bootstrap.cc:654] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.253573 13249 raft_consensus.cc:348] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.253816 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_aeb68c202cc642e6908287b378e38344 [id=d323713c509740a3ad2616eb8fe68337]
I20240506 19:57:02.253875 13249 raft_consensus.cc:374] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.254273 13249 raft_consensus.cc:729] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.254606 13249 consensus_queue.cc:260] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.254770 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3f897aef86e1464082e99df8a664489a [id=d38ac469f6724317ab026105379afa63]
I20240506 19:57:02.255858 13249 ts_tablet_manager.cc:1398] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:02.256549 13249 tablet_bootstrap.cc:492] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.247123 13245 consensus_queue.cc:260] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.260521 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c2e72287331646f2b30cf2f0e8d0509a [id=d3a119ec22a04657babb58caf69d7604]
I20240506 19:57:02.262426 12847 raft_consensus.cc:3037] T ff3049a9fa8049e5b84230e5e6e8245c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.263192 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6f20f6b912c249479dbf461093583817 [id=d401826620ea46ad8df6016c1f1cf968]
I20240506 19:57:02.267465 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_d61d010cfd0a45fc9c7413c61e051f1a [id=d4ec4f62a41b458faa3e28b41a2523cd]
I20240506 19:57:02.270773 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_112c674a2f6a46cb8e60669eb015488f [id=d50032a383514ade85a7e639245332b6]
I20240506 19:57:02.274144 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a2366efe03f84bdcabf1d6d62fc25ec7 [id=d5d0e57991d34f01822c91354809b6aa]
I20240506 19:57:02.274669 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_82ef5c37628b4923b2e9cce9d2ff4e88 [id=d64aa72e3aea448196fa09b04a0dfe7d]
I20240506 19:57:02.283951 12922 leader_election.cc:302] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.284569 14797 raft_consensus.cc:2781] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.284765 14797 raft_consensus.cc:686] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.285135 14797 consensus_queue.cc:237] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.286054 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_bc52ab9a06b149e999bf34c308e24839 [id=d8af7c79f3994f49b9bd7c11eacd65e6]
I20240506 19:57:02.279208 13245 ts_tablet_manager.cc:1398] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.034s user 0.000s sys 0.003s
I20240506 19:57:02.290477 12848 raft_consensus.cc:3037] T cd6245e4adce4038ad6f988877d75045 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.289394 13245 tablet_bootstrap.cc:492] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.304751 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_34e944977e794f829c328e4fc138baec [id=da72f500d27941dd826d6896b02cbeeb]
I20240506 19:57:02.305744 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_74ffa3bb4ed14490a8fcf4eb9e29a709 [id=dbcaca64a73545e78c668161948a1b48]
I20240506 19:57:02.306962 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_a5d469d01eb3446b81546610731630c3 [id=dcf79d1fdf7b4750a4ed2fae4200783e]
I20240506 19:57:02.307447 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_113b557a89234f5696e404ae3a3cc0e5 [id=dd3ff8db260242beba43003f552b628c]
I20240506 19:57:02.312129 12922 leader_election.cc:302] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.312799 12922 leader_election.cc:302] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.313153 14700 raft_consensus.cc:2781] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.313287 14700 raft_consensus.cc:482] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.313416 14700 raft_consensus.cc:3037] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.316128 14700 raft_consensus.cc:504] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.317662 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aae034f52645ba96827923f4fa305c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.317773 15011 raft_consensus.cc:2781] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.317951 15011 raft_consensus.cc:686] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.318214 15011 consensus_queue.cc:237] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:02.318426 12921 leader_election.cc:341] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.318645 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_540b5b87de2e447a8ce8a0b8f758cab7 [id=e094cbf306144ec3a64a14f9feb83085]
I20240506 19:57:02.319279 14700 leader_election.cc:288] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:02.320037 13058 consensus_peers.cc:485] T 11b00381c6aa42c9b969410b51ebe81e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.320912 13249 tablet_bootstrap.cc:654] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.331997 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f9f5211355fa4a10b23a184e7c55ae7c [id=e311f1c887124c15a887b9def426858e]
I20240506 19:57:02.333982 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eadf31733273444286cc81e91a890ddb [id=e341934744774a2eb8baf489109c16f2]
I20240506 19:57:02.335073 13245 tablet_bootstrap.cc:654] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.335806 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aae034f52645ba96827923f4fa305c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:02.336037 13124 raft_consensus.cc:3037] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.339401 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_4467002fce564e2fb50a2b9755805518 [id=e354c012edc64f49bdc0ab0dde85d05e]
I20240506 19:57:02.339577 13124 raft_consensus.cc:2454] T d0aae034f52645ba96827923f4fa305c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:02.340420 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_e71ba8bf0b874298b72cae27e8607c6c [id=e450843bf078436e83034537873af619]
I20240506 19:57:02.340934 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_82ad0ca5d8e2425b98dc9d534a3436cc [id=e4c63adb11504619b37bdc6c1e025c36]
I20240506 19:57:02.342023 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_ef03c2589fde409f95b535d84b66bf29 [id=e8385d8ad6714acf91d8cb6b0de6469c]
I20240506 19:57:02.352731 12922 leader_election.cc:302] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.353179 14700 raft_consensus.cc:2781] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.353534 14700 raft_consensus.cc:686] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.353845 14700 consensus_queue.cc:237] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.372758 14910 raft_consensus.cc:482] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.373013 14910 raft_consensus.cc:504] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.373896 14910 leader_election.cc:288] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:02.375774 12781 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:57:02.376309 13249 tablet_bootstrap.cc:492] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.376690 13249 ts_tablet_manager.cc:1367] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.000s
I20240506 19:57:02.378234 13249 raft_consensus.cc:348] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.378865 13249 raft_consensus.cc:374] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.379112 13249 raft_consensus.cc:729] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
W20240506 19:57:02.379807 12783 leader_election.cc:341] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: bd26e6b8115644cabe6532c7be1bbe95
I20240506 19:57:02.379920 12783 leader_election.cc:302] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.380365 14910 raft_consensus.cc:2726] T bd26e6b8115644cabe6532c7be1bbe95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:02.384644 13058 consensus_peers.cc:485] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.385105 15098 raft_consensus.cc:482] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.385264 15098 raft_consensus.cc:504] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.386111 15098 leader_election.cc:288] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.386610 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7ecd2c36503246d8992d73e00cf06129 [id=e96f8b4e07a6461095da74b4e260046b]
I20240506 19:57:02.386862 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.387120 12983 raft_consensus.cc:2454] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:02.387565 13057 leader_election.cc:302] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.387712 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:02.388027 15098 raft_consensus.cc:2781] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.388093 15098 raft_consensus.cc:482] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.388156 15098 raft_consensus.cc:3037] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.389839 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_66fa4fdbd72742d9aa51e8911f3b72cc [id=e9b16aee765e41a4898ad36b61832cbd]
I20240506 19:57:02.391113 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20 [id=e9d56f7b210b48caac7ad8b211e7de13]
I20240506 19:57:02.392848 15098 raft_consensus.cc:504] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.394606 15098 leader_election.cc:288] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.394618 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:02.394922 12986 raft_consensus.cc:3037] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.399919 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_5de7f6e4f9f5452291f85ec5db486f81 [id=ea7a610430e34ca2b3ab4868800722d5]
I20240506 19:57:02.400321 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.400563 12986 raft_consensus.cc:2454] T 744864043f9347e0a13f8c2a7d6b147f P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:02.400826 13245 tablet_bootstrap.cc:492] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.401021 13245 ts_tablet_manager.cc:1367] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.112s user 0.005s sys 0.000s
I20240506 19:57:02.401940 13057 leader_election.cc:302] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.402091 13245 raft_consensus.cc:348] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.402257 13245 raft_consensus.cc:374] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.402541 13245 raft_consensus.cc:729] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.402539 15098 raft_consensus.cc:2781] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.402838 15098 raft_consensus.cc:686] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:02.402745 13245 consensus_queue.cc:260] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.403348 15098 consensus_queue.cc:237] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.403645 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_0c23ca41c31d456e89fdd0046e87d6f9 [id=eac26d874c4747b0b7eb9f353ccd32a4]
I20240506 19:57:02.404284 13245 ts_tablet_manager.cc:1398] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:02.404415 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_cae804c3fdd14f67bde6f49878f4d248 [id=ebcb68f58ca747c7afb08c4c4a900685]
I20240506 19:57:02.404739 13245 tablet_bootstrap.cc:492] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:02.405889 13058 leader_election.cc:341] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:02.407065 13058 leader_election.cc:341] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.407565 13244 tablet_bootstrap.cc:492] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.407703 13245 tablet_bootstrap.cc:654] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.407721 13244 ts_tablet_manager.cc:1367] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.157s user 0.049s sys 0.089s
I20240506 19:57:02.408423 13244 raft_consensus.cc:348] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.408623 13244 raft_consensus.cc:374] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.408670 13244 raft_consensus.cc:729] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.408902 13244 consensus_queue.cc:260] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.409523 13244 ts_tablet_manager.cc:1398] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:02.409907 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_917d34debc2543bdb5c53f1cad733526 [id=eccaa60f1c9f49f0bc5176572b44ab76]
I20240506 19:57:02.410148 13244 tablet_bootstrap.cc:492] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.411936 13244 tablet_bootstrap.cc:654] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.412034 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9765ef1ea1284910aea85d1d0de68bb3 [id=ed62c20271fe4b58be6d1d66643da8a1]
I20240506 19:57:02.422943 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_b0f493ce68af4ecb9209de0432cb8184 [id=ed9b9d65276145bfaa26ab5ccc785b21]
I20240506 19:57:02.423539 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7297af847f20409aae62497c2b197328 [id=edf335649ed04b0a9027ba71825918ee]
I20240506 19:57:02.425902 13249 consensus_queue.cc:260] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.431093 13249 ts_tablet_manager.cc:1398] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.054s user 0.003s sys 0.000s
I20240506 19:57:02.432618 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_922f89e002cf480a9fca0e23f0eeb801 [id=ef089914d9514c29b2e3341c27e1a084]
I20240506 19:57:02.433859 13249 tablet_bootstrap.cc:492] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.439834 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f76cd2f478bd458697b5873a16fbb55b [id=ef23a1148e3648f49b43c73a89b6e598]
I20240506 19:57:02.440955 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_02ef9cd7cbba4ec1bc2971e5c8f43cda [id=ef25759917464cb5a3c0feb3f384c417]
I20240506 19:57:02.441354 14910 raft_consensus.cc:482] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.441506 14910 raft_consensus.cc:504] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.442144 14910 leader_election.cc:288] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.442960 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.444177 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f28c1f55fd14aca86cb00e43b3a6b76" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.444406 15104 consensus_queue.cc:1035] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.445319 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_114d9e27c6d74425802c46e8610d3343 [id=efe4c73c778d4128b0ff1a51a985f5c1]
I20240506 19:57:02.446112 13244 tablet_bootstrap.cc:492] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.446261 13244 ts_tablet_manager.cc:1367] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240506 19:57:02.446321 14910 consensus_queue.cc:1035] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.446424 13249 tablet_bootstrap.cc:654] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.447604 12781 leader_election.cc:302] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.448387 14910 raft_consensus.cc:2726] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.455492 15104 consensus_queue.cc:1035] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:57:02.455982 12921 consensus_peers.cc:485] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.470913 13245 tablet_bootstrap.cc:492] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.471097 13245 ts_tablet_manager.cc:1367] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.001s
I20240506 19:57:02.472275 13245 raft_consensus.cc:348] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.472625 13245 raft_consensus.cc:374] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.472748 13245 raft_consensus.cc:729] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.473084 13245 consensus_queue.cc:260] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.474871 13245 ts_tablet_manager.cc:1398] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240506 19:57:02.474867 15010 raft_consensus.cc:482] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.475037 15010 raft_consensus.cc:504] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.475749 15010 leader_election.cc:288] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.476475 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb87878d06417cbabf3c218728dbdc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:02.477246 12921 leader_election.cc:341] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.478600 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb87878d06417cbabf3c218728dbdc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.478775 13245 tablet_bootstrap.cc:492] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.478788 13124 raft_consensus.cc:2454] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:02.479738 12922 leader_election.cc:302] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.480126 15010 raft_consensus.cc:2781] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.480253 15010 raft_consensus.cc:482] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.480335 15010 raft_consensus.cc:3037] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.480432 13245 tablet_bootstrap.cc:654] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.482028 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_9e3f4bda34084998998a5a34a7717ffd [id=f031e266448b45adbdaaa7a84265da5c]
I20240506 19:57:02.481585 13244 raft_consensus.cc:348] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.482928 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_bdd8b0bb2f654920a49117008a662af1 [id=f0c703ff6122447b929a5e49ac7f3f42]
I20240506 19:57:02.483060 15010 raft_consensus.cc:504] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.484030 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb87878d06417cbabf3c218728dbdc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:02.484197 13124 raft_consensus.cc:3037] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.484385 15010 leader_election.cc:288] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.485056 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c364eee867924659ae35358d322b91ae [id=f22bf1138894495291b1ac38b34d0e6d]
I20240506 19:57:02.487061 13124 raft_consensus.cc:2454] T ffbb87878d06417cbabf3c218728dbdc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:02.488145 12922 leader_election.cc:302] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.488642 15010 raft_consensus.cc:2781] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.488790 15010 raft_consensus.cc:686] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.482609 13244 raft_consensus.cc:374] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.489122 15010 consensus_queue.cc:237] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.488487 14910 raft_consensus.cc:482] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.484122 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb87878d06417cbabf3c218728dbdc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.489620 14910 raft_consensus.cc:504] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:02.490082 12921 leader_election.cc:341] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.490201 14910 leader_election.cc:288] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:02.491107 12783 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f3a8531a1793404699532295164a3895
I20240506 19:57:02.491201 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a8531a1793404699532295164a3895" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.491457 12985 raft_consensus.cc:2454] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 0.
I20240506 19:57:02.489182 13244 raft_consensus.cc:729] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.491856 12781 leader_election.cc:302] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.492347 14910 raft_consensus.cc:2781] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.492503 14910 raft_consensus.cc:482] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.492700 14910 raft_consensus.cc:3037] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.493026 13244 consensus_queue.cc:260] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.497339 14910 raft_consensus.cc:504] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.498891 14910 leader_election.cc:288] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:02.499934 12783 leader_election.cc:341] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f3a8531a1793404699532295164a3895
I20240506 19:57:02.500563 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a8531a1793404699532295164a3895" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:02.500838 12985 raft_consensus.cc:3037] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:02.500865 12783 consensus_peers.cc:485] T 1d53f3daaa2949a2b6883435f031dc37 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d53f3daaa2949a2b6883435f031dc37. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.504153 12985 raft_consensus.cc:2454] T f3a8531a1793404699532295164a3895 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef18e88b87064ed4b6c20f9325ebcc33 in term 1.
I20240506 19:57:02.504621 12781 leader_election.cc:302] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.505200 14910 raft_consensus.cc:2781] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.505347 14910 raft_consensus.cc:686] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 LEADER]: Becoming Leader. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Running, Role: LEADER
I20240506 19:57:02.505632 14910 consensus_queue.cc:237] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:02.511955 12783 consensus_peers.cc:485] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d43da1d12ce647dcbcd31263932b2f54. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:02.536548 13058 consensus_peers.cc:485] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.536900 13245 tablet_bootstrap.cc:492] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.537470 13245 ts_tablet_manager.cc:1367] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20240506 19:57:02.540769 14910 raft_consensus.cc:482] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.541064 14910 raft_consensus.cc:504] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:02.542765 12783 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 85bc7cfd006f4d4ca95541dab6873d32
I20240506 19:57:02.542526 13245 raft_consensus.cc:348] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.543373 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f00049dcfdce4881891eed1bf5951d5b [id=f2b74839f0a0480f8d37f0d11f3743ea]
I20240506 19:57:02.543305 13245 raft_consensus.cc:374] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:57:02.543635 13058 consensus_peers.cc:485] T a568db06324a4fd1b58a29ef0831eb71 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.543648 13245 raft_consensus.cc:729] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.544112 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_44b86b8ac10f4b758b4356ccf982efb0 [id=f300837e10554f88bddd6e6e6bce630b]
I20240506 19:57:02.544432 14910 leader_election.cc:288] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.544580 13245 consensus_queue.cc:260] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.547225 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_07ee733aefe144be9fbd0d5a9cf3dbca [id=f301f42c7ec249c482779c6ac18a73a4]
I20240506 19:57:02.547220 13245 ts_tablet_manager.cc:1398] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:57:02.547906 13249 tablet_bootstrap.cc:492] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.548167 13249 ts_tablet_manager.cc:1367] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.115s user 0.022s sys 0.033s
I20240506 19:57:02.549094 13245 tablet_bootstrap.cc:492] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.549472 13249 raft_consensus.cc:348] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.549818 13249 raft_consensus.cc:374] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.549942 13249 raft_consensus.cc:729] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.550352 13249 consensus_queue.cc:260] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.551462 13249 ts_tablet_manager.cc:1398] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:02.559862 13245 tablet_bootstrap.cc:654] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.562528 13244 ts_tablet_manager.cc:1398] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.116s user 0.005s sys 0.000s
I20240506 19:57:02.562987 13244 tablet_bootstrap.cc:492] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.565361 13244 tablet_bootstrap.cc:654] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.565884 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_3270a7fb14b445ccbe0524f7b444e06d [id=f424bbb43b2a47449f998c21dae05cb9]
I20240506 19:57:02.567260 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_03cf2611bf48455a925b28b842f1e351 [id=f48f19c2b56e498dbdd65515f355c419]
W20240506 19:57:02.569901 13058 consensus_peers.cc:485] T 0fb9c05d703f417780e77359e19e542e P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.570308 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c4ab6be6ef2d46bdb824d78a7eea1f20 [id=f4cd8e862ab54bf3b10336065c65b7e8]
I20240506 19:57:02.573829 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8 [id=f5816273b8914052a9b58d5495bf45a1]
I20240506 19:57:02.575665 15011 consensus_queue.cc:1035] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.578228 13245 tablet_bootstrap.cc:492] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.578568 13245 ts_tablet_manager.cc:1367] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.002s
I20240506 19:57:02.579838 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_68a353433c044a28b362eb7d3606bfb6 [id=f5c22b4330f041eea15b7c14968d1732]
I20240506 19:57:02.580536 13245 raft_consensus.cc:348] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
W20240506 19:57:02.581082 12896 tablet.cc:2369] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:57:02.581094 13245 raft_consensus.cc:374] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.581219 13245 raft_consensus.cc:729] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.582190 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_f9d1e5113608496eb4aad3218eb44a65 [id=f7188aea4c354dd4920e4fd94f6b5323]
I20240506 19:57:02.581853 13245 consensus_queue.cc:260] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.582819 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_6b7f6251027641e399717012c63b5ac2 [id=f74234f1ff834fbd9f3e26d93cdb7665]
I20240506 19:57:02.583443 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a]
I20240506 19:57:02.584645 13245 ts_tablet_manager.cc:1398] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 19:57:02.586337 13245 tablet_bootstrap.cc:492] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:02.590250 12781 leader_election.cc:341] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 85bc7cfd006f4d4ca95541dab6873d32
I20240506 19:57:02.590392 12781 leader_election.cc:302] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.591320 15141 raft_consensus.cc:2726] T 85bc7cfd006f4d4ca95541dab6873d32 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.594807 13245 tablet_bootstrap.cc:654] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.597808 14656 consensus_queue.cc:1035] T b6fa6add23b8403ba5a0a65ac6752a5b P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:57:02.598349 12921 consensus_peers.cc:485] T 2b949c48806d45c3a74efbc8503c0bdb P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.598608 15142 raft_consensus.cc:482] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.598731 15142 raft_consensus.cc:504] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.599367 15142 leader_election.cc:288] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.600376 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.601848 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85365758f6a64536ab61e8349de63dea" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.602389 12781 leader_election.cc:302] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.602952 15142 raft_consensus.cc:2726] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.615279 13245 tablet_bootstrap.cc:492] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.615646 13245 ts_tablet_manager.cc:1367] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.003s
I20240506 19:57:02.617074 13245 raft_consensus.cc:348] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.617483 13245 raft_consensus.cc:374] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.617633 13245 raft_consensus.cc:729] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.619525 13245 consensus_queue.cc:260] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.621337 13245 ts_tablet_manager.cc:1398] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:02.622965 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_7b12e2d9a5d0496586f34ba591615326 [id=fab32d0c08604ab7976f3c3ef48e4859]
I20240506 19:57:02.623404 13244 tablet_bootstrap.cc:492] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.623625 13244 ts_tablet_manager.cc:1367] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.004s
I20240506 19:57:02.623929 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b]
I20240506 19:57:02.624393 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_61becbd043ee470ca2ee3f4969d59971 [id=fcfc5bd1da1b4625a2b0ebee1207abdd]
I20240506 19:57:02.624454 13244 raft_consensus.cc:348] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.624660 13244 raft_consensus.cc:374] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.624734 13244 raft_consensus.cc:729] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.624948 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_71a4779621224f0aa1c1b137ad0de871 [id=fd183a93acb14d58b828ecc56e2002d3]
I20240506 19:57:02.625031 13244 consensus_queue.cc:260] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.625957 13244 ts_tablet_manager.cc:1398] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:02.627491 13244 tablet_bootstrap.cc:492] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.628623 13245 tablet_bootstrap.cc:492] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.631711 13245 tablet_bootstrap.cc:654] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:02.632206 13036 tablet.cc:2369] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:57:02.636314 15141 raft_consensus.cc:482] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.636474 13249 tablet_bootstrap.cc:492] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.636466 15141 raft_consensus.cc:504] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.637375 15141 leader_election.cc:288] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.637820 13249 tablet_bootstrap.cc:654] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:02.639878 12783 leader_election.cc:341] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 17da01620b8b41008421b977455e0c9a
I20240506 19:57:02.640118 14656 consensus_queue.cc:1035] T df04fb90230d4bf4a060db993adafd14 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:57:02.641392 12781 leader_election.cc:341] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 17da01620b8b41008421b977455e0c9a
I20240506 19:57:02.641503 12781 leader_election.cc:302] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.641664 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_13e6e7d68e57433ebed02111652c43a9 [id=fe195172a40d45a5aef9388abdc14672]
I20240506 19:57:02.642496 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b]
I20240506 19:57:02.642737 15141 raft_consensus.cc:2726] T 17da01620b8b41008421b977455e0c9a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.655084 13244 tablet_bootstrap.cc:654] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.657217 13245 tablet_bootstrap.cc:492] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.657862 13245 ts_tablet_manager.cc:1367] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20240506 19:57:02.660100 13245 raft_consensus.cc:348] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.660590 13245 raft_consensus.cc:374] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.660780 13245 raft_consensus.cc:729] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.661175 13245 consensus_queue.cc:260] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.662824 13245 ts_tablet_manager.cc:1398] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:02.664156 13245 tablet_bootstrap.cc:492] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.669621 13245 tablet_bootstrap.cc:654] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.676509 13249 tablet_bootstrap.cc:492] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.676828 13249 ts_tablet_manager.cc:1367] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.005s
I20240506 19:57:02.678018 13249 raft_consensus.cc:348] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.678428 13249 raft_consensus.cc:374] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.678617 13249 raft_consensus.cc:729] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.678817 14859 catalog_manager.cc:669] Loaded metadata for table default.Table_eebcdd01355340739cb45d1acca7daa6 [id=ff67c807ba92484dbda8d03d7df13c6e]
I20240506 19:57:02.679121 13249 consensus_queue.cc:260] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.680843 13249 ts_tablet_manager.cc:1398] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:02.682075 13249 tablet_bootstrap.cc:492] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.693116 15011 raft_consensus.cc:482] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.693431 15011 raft_consensus.cc:504] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.694005 13244 tablet_bootstrap.cc:492] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.694875 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e22e5a015f74526b49363bc789cb4ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.695089 15011 leader_election.cc:288] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.695133 13124 raft_consensus.cc:2454] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:02.695618 12922 leader_election.cc:302] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.695914 15011 raft_consensus.cc:2781] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.696071 15011 raft_consensus.cc:482] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.696213 15011 raft_consensus.cc:3037] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.694923 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e22e5a015f74526b49363bc789cb4ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:02.697278 12921 leader_election.cc:341] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.697849 13244 ts_tablet_manager.cc:1367] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.001s
I20240506 19:57:02.699096 13244 raft_consensus.cc:348] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.700780 13244 raft_consensus.cc:374] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:57:02.702035 13178 tablet.cc:2369] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 19:57:02.702142 15011 raft_consensus.cc:504] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.702869 13244 raft_consensus.cc:729] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.708170 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e22e5a015f74526b49363bc789cb4ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:02.708350 13124 raft_consensus.cc:3037] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.708983 13249 tablet_bootstrap.cc:654] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.709107 15011 leader_election.cc:288] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.711056 13124 raft_consensus.cc:2454] T 1e22e5a015f74526b49363bc789cb4ed P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:02.711935 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e22e5a015f74526b49363bc789cb4ed" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:02.712745 12921 leader_election.cc:341] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.712975 12922 leader_election.cc:302] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.713259 15011 raft_consensus.cc:2781] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.713608 15011 raft_consensus.cc:686] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.714430 12848 raft_consensus.cc:3037] T ecc88443c04b42d19cc7fba339eeeaf5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.714443 13245 tablet_bootstrap.cc:492] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.715832 13245 ts_tablet_manager.cc:1367] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.012s
I20240506 19:57:02.714329 15011 consensus_queue.cc:237] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.703682 13244 consensus_queue.cc:260] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.720170 12847 raft_consensus.cc:3037] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.720746 13244 ts_tablet_manager.cc:1398] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240506 19:57:02.722805 13244 tablet_bootstrap.cc:492] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.723170 13245 raft_consensus.cc:348] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.725566 13245 raft_consensus.cc:374] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.726320 13245 raft_consensus.cc:729] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.728480 13245 consensus_queue.cc:260] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.733201 13245 ts_tablet_manager.cc:1398] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
I20240506 19:57:02.733470 13244 tablet_bootstrap.cc:654] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.735401 13245 tablet_bootstrap.cc:492] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.743511 13245 tablet_bootstrap.cc:654] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.745677 14915 consensus_queue.cc:1035] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:57:02.749347 13058 consensus_peers.cc:485] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.758200 13245 tablet_bootstrap.cc:492] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.759099 13245 ts_tablet_manager.cc:1367] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240506 19:57:02.761052 14700 consensus_queue.cc:1035] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.765362 13245 raft_consensus.cc:348] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.766587 13245 raft_consensus.cc:374] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.767072 13245 raft_consensus.cc:729] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.769038 15011 raft_consensus.cc:482] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.769214 15011 raft_consensus.cc:504] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.768321 13245 consensus_queue.cc:260] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.770221 12842 raft_consensus.cc:3037] T 0f28c1f55fd14aca86cb00e43b3a6b76 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.772188 13245 ts_tablet_manager.cc:1398] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
W20240506 19:57:02.769465 12921 consensus_peers.cc:485] T 5b375490272a4dec9a743f6458f6d6de P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.773900 15141 raft_consensus.cc:482] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.774055 15141 raft_consensus.cc:504] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.774192 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f078559df48f4d23869ddcb4b3fca9fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.774040 13245 tablet_bootstrap.cc:492] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.774510 13124 raft_consensus.cc:2454] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:02.774689 15141 leader_election.cc:288] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.774848 12922 leader_election.cc:302] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.775130 14700 raft_consensus.cc:2781] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.775203 14700 raft_consensus.cc:482] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.775285 14700 raft_consensus.cc:3037] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:02.776355 12783 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2ff3435c11854649b7daf74f3d8c33d7
I20240506 19:57:02.778772 15141 raft_consensus.cc:482] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.778910 15141 raft_consensus.cc:504] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.779497 15141 leader_election.cc:288] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.779418 14700 raft_consensus.cc:504] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:02.780455 12783 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 116df9bd6ab04a30ba09861e765d7267
I20240506 19:57:02.780525 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f078559df48f4d23869ddcb4b3fca9fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:02.781080 12921 leader_election.cc:341] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.781262 14700 leader_election.cc:288] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.782451 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f078559df48f4d23869ddcb4b3fca9fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:02.782634 13120 raft_consensus.cc:3037] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.783310 13244 tablet_bootstrap.cc:492] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.783584 13244 ts_tablet_manager.cc:1367] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.000s
I20240506 19:57:02.784111 13245 tablet_bootstrap.cc:654] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:02.784663 12781 leader_election.cc:341] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2ff3435c11854649b7daf74f3d8c33d7
I20240506 19:57:02.784781 12781 leader_election.cc:302] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.785215 13120 raft_consensus.cc:2454] T f078559df48f4d23869ddcb4b3fca9fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:57:02.785372 12781 leader_election.cc:341] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 116df9bd6ab04a30ba09861e765d7267
I20240506 19:57:02.785457 12781 leader_election.cc:302] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.785625 15141 raft_consensus.cc:2726] T 2ff3435c11854649b7daf74f3d8c33d7 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.785969 15141 raft_consensus.cc:2726] T 116df9bd6ab04a30ba09861e765d7267 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.784767 13244 raft_consensus.cc:348] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.786193 13244 raft_consensus.cc:374] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:57:02.786144 12922 consensus_peers.cc:485] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 373e04e93a5543f9934077006eaf2ba9. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.786424 13244 raft_consensus.cc:729] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.786466 12922 leader_election.cc:302] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.786732 15009 raft_consensus.cc:2781] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.786871 15009 raft_consensus.cc:686] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.786883 13244 consensus_queue.cc:260] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.788128 13244 ts_tablet_manager.cc:1398] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:57:02.788846 15009 consensus_queue.cc:237] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.790153 13244 tablet_bootstrap.cc:492] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.792526 15011 leader_election.cc:288] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:02.792928 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f078559df48f4d23869ddcb4b3fca9fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:02.793434 12921 leader_election.cc:341] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.798283 13245 tablet_bootstrap.cc:492] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.799139 13245 ts_tablet_manager.cc:1367] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20240506 19:57:02.800853 13244 tablet_bootstrap.cc:654] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.804976 15011 consensus_queue.cc:1035] T 4b4ad903e75547fa829b82cc722fef1e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.804648 13245 raft_consensus.cc:348] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.807005 13245 raft_consensus.cc:374] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.807457 13245 raft_consensus.cc:729] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.807991 15141 consensus_queue.cc:1035] T d43da1d12ce647dcbcd31263932b2f54 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.808703 13245 consensus_queue.cc:260] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.813222 13245 ts_tablet_manager.cc:1398] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240506 19:57:02.814071 14797 consensus_queue.cc:1035] T 373e04e93a5543f9934077006eaf2ba9 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:02.815347 13245 tablet_bootstrap.cc:492] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.828516 13245 tablet_bootstrap.cc:654] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.832540 13244 tablet_bootstrap.cc:492] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.833504 13244 ts_tablet_manager.cc:1367] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20240506 19:57:02.835263 13244 raft_consensus.cc:348] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.835676 13244 raft_consensus.cc:374] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.835819 13244 raft_consensus.cc:729] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.836263 13244 consensus_queue.cc:260] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.837695 13244 ts_tablet_manager.cc:1398] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:02.839243 13244 tablet_bootstrap.cc:492] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:02.842204 14081 leader_election.cc:334] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819): Timed out: RequestConsensusVote RPC to 127.12.48.253:36819 timed out after 0.552s (SENT)
I20240506 19:57:02.842371 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:57:02.853206 13245 tablet_bootstrap.cc:492] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.854079 13245 ts_tablet_manager.cc:1367] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.000s
I20240506 19:57:02.855258 15142 raft_consensus.cc:482] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 059a5c4a3f7e49ef95af4b8341702745)
I20240506 19:57:02.855418 15142 raft_consensus.cc:504] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.856290 15142 leader_election.cc:288] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.856977 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b625c2bbd89746bc90b8c4c2483b3741" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.858039 15142 raft_consensus.cc:482] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.858157 15142 raft_consensus.cc:504] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.858688 15142 leader_election.cc:288] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:02.859598 12783 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:57:02.860169 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b625c2bbd89746bc90b8c4c2483b3741" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.859383 13245 raft_consensus.cc:348] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.861181 12781 leader_election.cc:302] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.860807 13245 raft_consensus.cc:374] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240506 19:57:02.861593 12781 leader_election.cc:341] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: ceeae70e0d1144ea9e3e0f8727c15e6b
I20240506 19:57:02.861753 12781 leader_election.cc:302] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.861622 13245 raft_consensus.cc:729] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.862007 15141 raft_consensus.cc:2726] T b625c2bbd89746bc90b8c4c2483b3741 P ef18e88b87064ed4b6c20f9325ebcc33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 19:57:02.862146 15141 raft_consensus.cc:2726] T ceeae70e0d1144ea9e3e0f8727c15e6b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.863135 13245 consensus_queue.cc:260] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.867311 13249 tablet_bootstrap.cc:492] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.867590 13249 ts_tablet_manager.cc:1367] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.186s user 0.003s sys 0.029s
I20240506 19:57:02.869204 15141 raft_consensus.cc:482] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.869390 15141 raft_consensus.cc:504] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.869232 13249 raft_consensus.cc:348] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.869693 13249 raft_consensus.cc:374] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.869837 13249 raft_consensus.cc:729] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.870050 15141 leader_election.cc:288] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.870311 13249 consensus_queue.cc:260] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.872975 13249 ts_tablet_manager.cc:1398] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:57:02.873035 13245 ts_tablet_manager.cc:1398] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240506 19:57:02.875567 13249 tablet_bootstrap.cc:492] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
W20240506 19:57:02.871341 12783 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2b3268094cbf481698cb373c1bf6dfba
W20240506 19:57:02.875912 12781 leader_election.cc:341] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2b3268094cbf481698cb373c1bf6dfba
I20240506 19:57:02.876428 12781 leader_election.cc:302] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.876780 15141 raft_consensus.cc:2726] T 2b3268094cbf481698cb373c1bf6dfba P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.870285 13244 tablet_bootstrap.cc:654] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.879904 12842 raft_consensus.cc:3037] T 85365758f6a64536ab61e8349de63dea P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.880915 13249 tablet_bootstrap.cc:654] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.887423 15011 raft_consensus.cc:482] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.887745 15011 raft_consensus.cc:504] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.889410 15011 leader_election.cc:288] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:02.889446 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f574aec984b83ab596e92a6f323fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.889701 13116 raft_consensus.cc:2454] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:02.889932 13245 tablet_bootstrap.cc:492] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.890368 12922 leader_election.cc:302] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:02.890805 15011 raft_consensus.cc:2781] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:02.891178 15011 raft_consensus.cc:482] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:02.891510 15011 raft_consensus.cc:3037] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.893119 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f574aec984b83ab596e92a6f323fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:02.894035 12921 leader_election.cc:341] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.895371 14859 tablet_loader.cc:96] loaded metadata for tablet 007f9df7ce8e40a495a4eacb7a026b00 (table default.Table_a5d469d01eb3446b81546610731630c3 [id=dcf79d1fdf7b4750a4ed2fae4200783e])
I20240506 19:57:02.896858 15011 raft_consensus.cc:504] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.879132 15176 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 19:57:02.898276 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f574aec984b83ab596e92a6f323fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:02.898909 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03f574aec984b83ab596e92a6f323fc" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
W20240506 19:57:02.898969 12921 leader_election.cc:341] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:02.899088 13116 raft_consensus.cc:3037] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.899292 15011 leader_election.cc:288] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:02.901885 13116 raft_consensus.cc:2454] T a03f574aec984b83ab596e92a6f323fc P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:02.901966 14859 tablet_loader.cc:96] loaded metadata for tablet 01f0def08b17403180b9b876c26428f7 (table default.Table_a6709e9c0a394e43a00065cca48b8cdc [id=b8163f6cb22c4d4aad204a87c727ed02])
I20240506 19:57:02.902324 12922 leader_election.cc:302] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:02.902854 15011 raft_consensus.cc:2781] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:02.903759 15011 raft_consensus.cc:686] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:02.904526 15011 consensus_queue.cc:237] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.904937 14859 tablet_loader.cc:96] loaded metadata for tablet 034b06c762cb4012ab7dc09ba7f1c41d (table default.Table_b820628699774953a9a7b655edefe41c [id=362f73af54c04bbc9de5657a30242626])
I20240506 19:57:02.914238 14859 tablet_loader.cc:96] loaded metadata for tablet 041612b3636a49db9e303877964a31ff (table default.Table_dbd73da3dbb94cc59750f8b4d27a46ee [id=a3a24a54b336458d8e6c2014ab3b38f8])
I20240506 19:57:02.915078 14859 tablet_loader.cc:96] loaded metadata for tablet 054d48760d684adeb8196b0532b07101 (table default.Table_bb460b3bc3c6433aa0d6abe2f3eea2e2 [id=80efe594c8b64a04864ac3a11061edd8])
I20240506 19:57:02.916699 14859 tablet_loader.cc:96] loaded metadata for tablet 060cfeedba024cebb19ccccb49d4d582 (table default.Table_f9d1e5113608496eb4aad3218eb44a65 [id=f7188aea4c354dd4920e4fd94f6b5323])
I20240506 19:57:02.920949 14859 tablet_loader.cc:96] loaded metadata for tablet 06234a1df4264ddd9abbcaf88292795f (table default.Table_fd059b81c3d045c48dbba6b7f1f2e31d [id=d2e2ea2858eb4b39b082df599d497112])
I20240506 19:57:02.921237 13249 tablet_bootstrap.cc:492] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:02.921532 13249 ts_tablet_manager.cc:1367] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.003s
I20240506 19:57:02.922174 13245 tablet_bootstrap.cc:654] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.922725 13249 raft_consensus.cc:348] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.924043 13249 raft_consensus.cc:374] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.924247 13249 raft_consensus.cc:729] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.924870 13249 consensus_queue.cc:260] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.926617 14859 tablet_loader.cc:96] loaded metadata for tablet 07a24a78936e4da68a4af40601b98e96 (table default.Table_3bd43b5745da4ad1ab82dceeec1f39b6 [id=38092479ce4249f5aed0ab679d4374ac])
I20240506 19:57:02.927462 14859 tablet_loader.cc:96] loaded metadata for tablet 07c7c7b2da9e403f8241a512e210c4d9 (table default.Table_54d5d8d8437a44a4a66e103564486b0d [id=6fd679db05504418b8f453a92ecf8397])
I20240506 19:57:02.928128 13249 ts_tablet_manager.cc:1398] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:57:02.928697 14859 tablet_loader.cc:96] loaded metadata for tablet 09a396f5c8684be7967be14b1b15195b (table default.Table_bec645fde8cf480cb56580e5edcd5d76 [id=37c6d37b352144daaa7f56ae07c6392e])
I20240506 19:57:02.929224 13249 tablet_bootstrap.cc:492] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:02.931013 14859 tablet_loader.cc:96] loaded metadata for tablet 0a6db7d0ad024f53ad76920bcbc0248e (table default.Table_28ea2a7143b84c1c9eeb79da8bf201ad [id=44207be9fc564b5bba7a4513e0b3c1bd])
I20240506 19:57:02.933562 14859 tablet_loader.cc:96] loaded metadata for tablet 0ad14a17520a49beb3682b8fd04f9e79 (table default.Table_cae804c3fdd14f67bde6f49878f4d248 [id=ebcb68f58ca747c7afb08c4c4a900685])
I20240506 19:57:02.934091 14859 tablet_loader.cc:96] loaded metadata for tablet 0b899244c35b4bbe9e916c778111f0c0 (table default.Table_dafad97b193f40038dfd516c93b79e96 [id=1a1d40c83b0744689cb8a85164109e0e])
I20240506 19:57:02.934516 14859 tablet_loader.cc:96] loaded metadata for tablet 0b90b1433a0b4cdfa53316f69ff19085 (table default.Table_7ecd2c36503246d8992d73e00cf06129 [id=e96f8b4e07a6461095da74b4e260046b])
I20240506 19:57:02.936146 14859 tablet_loader.cc:96] loaded metadata for tablet 0ce0a5be54ee4bafae89d59d29b0c8df (table default.Table_3eff05dacdb8400894bde49d2337d7f3 [id=a332c12a7dca471eb9e79b7974ea26a4])
I20240506 19:57:02.936764 14859 tablet_loader.cc:96] loaded metadata for tablet 0d29c7c9c933414fb0ca6e6d370eac62 (table default.Table_1b1ba78c0a194d509cee193f3bec562e [id=af9eea916c5f44b2a0c650481873e4f9])
I20240506 19:57:02.937261 14859 tablet_loader.cc:96] loaded metadata for tablet 0d8b7c300d574b539169209f84820326 (table default.Table_c2d2254e8afc4f8a80b105ceffab0b47 [id=3614ed5dd41b467aa26de18b1b04ed08])
I20240506 19:57:02.937716 14859 tablet_loader.cc:96] loaded metadata for tablet 0d94e3976a824bb9b10607d02764c8cc (table default.Table_a8f8c4d749444684a5b327c80c5f5829 [id=3e6deafd817f43c0b137bae39fb1d39c])
I20240506 19:57:02.938691 14859 tablet_loader.cc:96] loaded metadata for tablet 0ee1a4a3f060407ca9c33467474e4489 (table default.Table_a47ab384cae5499183603602a149e649 [id=28724ef5aa3b4dc2b399336e8959cf34])
I20240506 19:57:02.939167 14859 tablet_loader.cc:96] loaded metadata for tablet 0ef799f2e474413d8f76f220a6bda665 (table default.Table_2fda57476820461185abfb4e54f9a3d9 [id=893f3cabb2d84a349345cc6fbf9f4656])
I20240506 19:57:02.940388 14859 tablet_loader.cc:96] loaded metadata for tablet 10ca9a1eaffc45f486d7a7e937a3e787 (table default.Table_e87268955e8247709b420f45ae2e4e5d [id=12a8e13309bc412ba8404129577b3444])
I20240506 19:57:02.941628 14859 tablet_loader.cc:96] loaded metadata for tablet 11b00381c6aa42c9b969410b51ebe81e (table default.Table_da8cee75a0c844e1a4956550119cc252 [id=cf1b4de7eeaa4aabaa3dc7c5160ec86d])
W20240506 19:57:02.925835 13058 consensus_peers.cc:485] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.942356 14859 tablet_loader.cc:96] loaded metadata for tablet 1353e75a36d749fd8730744f88268742 (table default.Table_b86c2104b98d4723bfde3500950e640b [id=4d658c76e1e842588a67621ff1d945a0])
I20240506 19:57:02.942423 13244 tablet_bootstrap.cc:492] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.942564 13244 ts_tablet_manager.cc:1367] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.104s user 0.007s sys 0.000s
I20240506 19:57:02.943321 13244 raft_consensus.cc:348] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.943514 13244 raft_consensus.cc:374] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.943516 14859 tablet_loader.cc:96] loaded metadata for tablet 14cffcd4198846ddadae19cd5e80b318 (table default.Table_a010c4cc7a5348d79e8c910419d72c98 [id=1ee0a9b9312f4deda04f6199c1014af5])
I20240506 19:57:02.943576 13244 raft_consensus.cc:729] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.943765 13244 consensus_queue.cc:260] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.943928 14859 tablet_loader.cc:96] loaded metadata for tablet 154e6d7c8f72480fa36fe542a5a67706 (table default.Table_3482045609cb42209a8222185144e10f [id=66cd489407394793a3016a8b41c2e62a])
I20240506 19:57:02.944311 13244 ts_tablet_manager.cc:1398] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:02.944578 13244 tablet_bootstrap.cc:492] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.954345 13244 tablet_bootstrap.cc:654] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.955621 12843 raft_consensus.cc:3037] T de26793c006440dc9aecc07963ddb85e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:02.960495 15141 raft_consensus.cc:482] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.960642 15141 raft_consensus.cc:504] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:02.961244 15141 leader_election.cc:288] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.961830 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d09eafa82a0423a8358958e78f728e2" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:02.963035 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d09eafa82a0423a8358958e78f728e2" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:02.963637 12781 leader_election.cc:302] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.964079 15141 raft_consensus.cc:2726] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.967427 15141 raft_consensus.cc:482] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:02.967561 15141 raft_consensus.cc:504] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:02.967669 12921 consensus_peers.cc:485] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:02.968168 15141 leader_election.cc:288] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:02.968497 13249 tablet_bootstrap.cc:654] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:02.969475 12783 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 234e4f3439e84b5fbb2f7bab1dbd0822
W20240506 19:57:02.970861 12781 leader_election.cc:341] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 234e4f3439e84b5fbb2f7bab1dbd0822
I20240506 19:57:02.970957 12781 leader_election.cc:302] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:02.971285 15141 raft_consensus.cc:2726] T 234e4f3439e84b5fbb2f7bab1dbd0822 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:02.971511 13244 tablet_bootstrap.cc:492] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:02.971799 13244 ts_tablet_manager.cc:1367] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240506 19:57:02.973697 13244 raft_consensus.cc:348] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.974026 13244 raft_consensus.cc:374] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.974117 13244 raft_consensus.cc:729] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.974567 13244 consensus_queue.cc:260] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:02.975396 13245 tablet_bootstrap.cc:492] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:02.975636 13245 ts_tablet_manager.cc:1367] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.002s
I20240506 19:57:02.975958 13244 ts_tablet_manager.cc:1398] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:57:02.976794 13244 tablet_bootstrap.cc:492] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:02.977022 13245 raft_consensus.cc:348] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:02.977387 13245 raft_consensus.cc:374] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:02.977475 13245 raft_consensus.cc:729] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:02.977761 13245 consensus_queue.cc:260] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
W20240506 19:57:02.979106 13442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.254:33495, user_credentials={real_user=slave}} blocked reactor thread for 52953.8us
I20240506 19:57:02.979112 13245 ts_tablet_manager.cc:1398] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240506 19:57:02.980851 13442 consensus_peers.cc:485] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 -> Peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Couldn't send request to peer 190a2eabb30d4154960a44eb50a8a87d. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240506 19:57:02.982012 13245 tablet_bootstrap.cc:492] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:02.982998 14859 tablet_loader.cc:96] loaded metadata for tablet 176c61208b1c4518820a829d9f5c0a6f (table default.Table_c5b5c0c587d6447eb89d9677b41b4757 [id=54457b4758cf4a5b8f7a55ef6b21f741])
I20240506 19:57:02.984534 14859 tablet_loader.cc:96] loaded metadata for tablet 1893a7de2d52475db1baeb4e816b549b (table default.Table_9277b5f5bbf44d878345b1028267ba8a [id=274ffbb5e513489f85dcc6097c8cc444])
I20240506 19:57:02.985018 14859 tablet_loader.cc:96] loaded metadata for tablet 196b339d0bc54426b69a60fe59f1af00 (table default.Table_0bcaa070661f4dc28a869fe16f7bc6c0 [id=7e9605522cd24b86adeec813cb0d7697])
I20240506 19:57:02.985353 13244 tablet_bootstrap.cc:654] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:02.994128 14915 consensus_queue.cc:1035] T 44ce8749846847108edf5f554139cbd9 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.008963 13245 tablet_bootstrap.cc:654] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.017783 14915 raft_consensus.cc:482] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.017956 14915 raft_consensus.cc:504] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.018790 14915 leader_election.cc:288] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.019501 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd12a9cc3924f3eb8aec502e9fa2d47" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.020022 13058 leader_election.cc:341] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.020347 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd12a9cc3924f3eb8aec502e9fa2d47" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:57:03.020838 13057 leader_election.cc:341] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.020947 13057 leader_election.cc:302] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.021348 14915 raft_consensus.cc:2726] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.022046 13249 tablet_bootstrap.cc:492] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.022290 13249 ts_tablet_manager.cc:1367] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.093s user 0.000s sys 0.005s
I20240506 19:57:03.023610 13249 raft_consensus.cc:348] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.024010 13249 raft_consensus.cc:374] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.024113 13249 raft_consensus.cc:729] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.024798 13249 consensus_queue.cc:260] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.026788 13249 ts_tablet_manager.cc:1398] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240506 19:57:03.027949 13249 tablet_bootstrap.cc:492] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.028707 13244 tablet_bootstrap.cc:492] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:03.028885 13244 ts_tablet_manager.cc:1367] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.003s
I20240506 19:57:03.029688 13244 raft_consensus.cc:348] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.029901 13244 raft_consensus.cc:374] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.030006 13244 raft_consensus.cc:729] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.030236 13244 consensus_queue.cc:260] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.031572 13244 ts_tablet_manager.cc:1398] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:03.031788 14859 tablet_loader.cc:96] loaded metadata for tablet 1b7d4f86697c4ed88a9bdf065ed293ba (table default.Table_9a8bd42081bd47fdb4e31ba99d96756e [id=cefd5538663b400aaf74cbe5e2d56673])
I20240506 19:57:03.032810 14859 tablet_loader.cc:96] loaded metadata for tablet 1bb1559fe135498f9fefbbf04daa6909 (table default.Table_7604b11105d14a889c9b95c21133e402 [id=617fd0a3b26f40abaee4b419ead18cba])
I20240506 19:57:03.033718 13249 tablet_bootstrap.cc:654] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.034343 13244 tablet_bootstrap.cc:492] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:03.041095 13244 tablet_bootstrap.cc:654] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.033257 14859 tablet_loader.cc:96] loaded metadata for tablet 1c0ac966a105410b8d1b965eb2ad6082 (table default.Table_ad11ec97e2384012981d0daae3a8adbd [id=6921d75de60f46e087367fd1b28ce12d])
I20240506 19:57:03.045049 14859 tablet_loader.cc:96] loaded metadata for tablet 1cc57eb8ac864bfa803908412530e7a3 (table default.Table_c602a53461334dcca2cdcebd0af292cc [id=fc65f2f56f084d0497a4eec319f45d4b])
I20240506 19:57:03.045548 14700 consensus_queue.cc:1035] T 01c32eba2031446ab419cac64068068e P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.051988 14859 tablet_loader.cc:96] loaded metadata for tablet 1f5294840fa14a3d88ed1fe297601d1b (table default.Table_11d89ae7590b466eacd8a986b0d180a4 [id=3cd8fd373b3e40caa343e05139cacd74])
I20240506 19:57:03.052402 14859 tablet_loader.cc:96] loaded metadata for tablet 1ff9ad5381de45cc8d7cc9587566ef54 (table default.Table_ff4a6a2744944a95aae0f535c062d338 [id=72698ebd6fc5442099795eeb11cc028b])
I20240506 19:57:03.053783 13245 tablet_bootstrap.cc:492] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.054133 13245 ts_tablet_manager.cc:1367] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.002s
I20240506 19:57:03.055050 13245 raft_consensus.cc:348] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.055454 13245 raft_consensus.cc:374] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.055611 13245 raft_consensus.cc:729] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.056025 13245 consensus_queue.cc:260] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.058178 13245 ts_tablet_manager.cc:1398] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:03.062829 14859 tablet_loader.cc:96] loaded metadata for tablet 20d071e027174b03b01cf788c0b95e89 (table default.Table_b75f9589d33c4ca6b64d52bbf4bf7bca [id=a867a0cd1eca411a85db8f03ee181d2e])
I20240506 19:57:03.062947 14700 raft_consensus.cc:482] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.063109 14700 raft_consensus.cc:504] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.063828 14700 leader_election.cc:288] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.064527 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a396afff3b243ef84cb3dc3836fcdfd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.064527 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a396afff3b243ef84cb3dc3836fcdfd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.064757 13116 raft_consensus.cc:2454] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:03.065152 12922 leader_election.cc:302] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:03.064805 14859 tablet_loader.cc:96] loaded metadata for tablet 21059ce22bff4ccc8ae7d0de0c2fbd7a (table default.Table_960b875a518745a38b1a1e6699ec3a2f [id=2efd55b739244f81a16c345e3d8746af])
W20240506 19:57:03.065325 12921 leader_election.cc:341] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.065486 14700 raft_consensus.cc:2781] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.065565 14700 raft_consensus.cc:482] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.065649 14700 raft_consensus.cc:3037] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.066056 14859 tablet_loader.cc:96] loaded metadata for tablet 211497cd9802473da3f4ea40b40830c6 (table default.Table_204da63924eb46ad9044cbc009296da3 [id=537838979f76429da504e67aa7485fcb])
I20240506 19:57:03.066368 15142 raft_consensus.cc:482] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.066517 15142 raft_consensus.cc:504] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.067149 14859 tablet_loader.cc:96] loaded metadata for tablet 221e3b5120494336822036588cf65456 (table default.Table_79eff2000c124513b48aeefee23bb9a3 [id=8b305bf0e98344d1a63fe0c84e5ee0fe])
I20240506 19:57:03.067276 15142 leader_election.cc:288] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:57:03.068773 12783 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: a1160ab972124630a9be7abeefb4aa56
W20240506 19:57:03.068895 12781 leader_election.cc:341] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: a1160ab972124630a9be7abeefb4aa56
I20240506 19:57:03.068960 12783 leader_election.cc:302] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.069461 15142 raft_consensus.cc:2726] T a1160ab972124630a9be7abeefb4aa56 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.070856 13245 tablet_bootstrap.cc:492] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.070865 14859 tablet_loader.cc:96] loaded metadata for tablet 2271dc77d307453683ceb193bd2fd71d (table default.Table_aff3a2a3bcf74711a29fc36b829be54d [id=77bf1f6b60274489b239a5270fb3d76b])
I20240506 19:57:03.071756 14915 raft_consensus.cc:482] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.071875 14915 raft_consensus.cc:504] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.072489 14915 leader_election.cc:288] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.073158 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cb71b5962f4561bf0fdbe12bf45296" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.073798 13244 tablet_bootstrap.cc:492] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
W20240506 19:57:03.073869 13058 leader_election.cc:341] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.073959 13244 ts_tablet_manager.cc:1367] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.002s
I20240506 19:57:03.074618 15011 raft_consensus.cc:482] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.076258 13244 raft_consensus.cc:348] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.076516 13244 raft_consensus.cc:374] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.076611 13244 raft_consensus.cc:729] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.076876 13244 consensus_queue.cc:260] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.077597 13244 ts_tablet_manager.cc:1398] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:03.074764 15011 raft_consensus.cc:504] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.078768 15011 leader_election.cc:288] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.078994 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cb71b5962f4561bf0fdbe12bf45296" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.079167 12986 raft_consensus.cc:2454] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.079656 13057 leader_election.cc:302] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.080065 15104 raft_consensus.cc:2781] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.080142 15104 raft_consensus.cc:482] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.080205 15104 raft_consensus.cc:3037] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.080957 14700 raft_consensus.cc:504] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.081120 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd12a9cc3924f3eb8aec502e9fa2d47" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.081350 13116 raft_consensus.cc:2454] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:03.081918 12922 leader_election.cc:302] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:03.082343 14797 raft_consensus.cc:2781] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.082423 14797 raft_consensus.cc:482] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.082489 14797 raft_consensus.cc:3037] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.082579 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a396afff3b243ef84cb3dc3836fcdfd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:03.079229 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd12a9cc3924f3eb8aec502e9fa2d47" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.082782 13116 raft_consensus.cc:3037] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.082993 13244 tablet_bootstrap.cc:492] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:03.084671 14859 tablet_loader.cc:96] loaded metadata for tablet 22b9553636ec4943a045b6ae8017f7cc (table default.Table_eb945644f4344376aa5059f1a8223ce9 [id=18e06c57bec942d9b0c1339852f02dd1])
I20240506 19:57:03.085115 14797 raft_consensus.cc:504] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.085599 15098 raft_consensus.cc:482] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.085718 15098 raft_consensus.cc:504] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
W20240506 19:57:03.086031 12921 leader_election.cc:341] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.086073 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a396afff3b243ef84cb3dc3836fcdfd" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.086282 15098 leader_election.cc:288] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:03.087059 12921 leader_election.cc:341] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.088234 14797 leader_election.cc:288] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.088932 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd12a9cc3924f3eb8aec502e9fa2d47" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.089071 14700 leader_election.cc:288] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.089519 13116 raft_consensus.cc:2454] T 7a396afff3b243ef84cb3dc3836fcdfd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:03.089460 15104 raft_consensus.cc:504] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.090071 15104 leader_election.cc:288] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.090099 12922 leader_election.cc:302] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.090114 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd12a9cc3924f3eb8aec502e9fa2d47" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:03.090246 13116 raft_consensus.cc:3037] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.090843 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cb71b5962f4561bf0fdbe12bf45296" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.091176 13245 tablet_bootstrap.cc:654] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.091470 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cb71b5962f4561bf0fdbe12bf45296" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:03.091630 12986 raft_consensus.cc:3037] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.093432 12921 leader_election.cc:341] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.093479 13116 raft_consensus.cc:2454] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
W20240506 19:57:03.093585 13058 leader_election.cc:341] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.093967 12922 leader_election.cc:302] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.094599 14797 raft_consensus.cc:2781] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.094748 15011 raft_consensus.cc:2781] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.094831 15011 raft_consensus.cc:686] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:03.095036 15011 consensus_queue.cc:237] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.095333 12986 raft_consensus.cc:2454] T 09cb71b5962f4561bf0fdbe12bf45296 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:03.095732 13057 leader_election.cc:302] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.095999 15098 raft_consensus.cc:2781] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.096109 15098 raft_consensus.cc:686] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:03.096319 15098 consensus_queue.cc:237] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.097076 14797 raft_consensus.cc:686] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:03.097347 14797 consensus_queue.cc:237] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.103216 13245 tablet_bootstrap.cc:492] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.103523 13245 ts_tablet_manager.cc:1367] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240506 19:57:03.105448 13245 raft_consensus.cc:348] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.106035 13245 raft_consensus.cc:374] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.106215 13245 raft_consensus.cc:729] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.106675 13245 consensus_queue.cc:260] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.108072 13245 ts_tablet_manager.cc:1398] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:57:03.110530 13245 tablet_bootstrap.cc:492] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:03.113875 13058 consensus_peers.cc:485] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.117152 13249 tablet_bootstrap.cc:492] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.118173 13249 ts_tablet_manager.cc:1367] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.005s
I20240506 19:57:03.119513 13245 tablet_bootstrap.cc:654] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.088835 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9944e5bfe0a741d69b8f9b6a9dad84dd" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.121220 13058 leader_election.cc:341] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.095885 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9944e5bfe0a741d69b8f9b6a9dad84dd" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.121699 12985 raft_consensus.cc:2454] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.122314 13057 leader_election.cc:302] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.122633 15098 raft_consensus.cc:2781] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.122830 15098 raft_consensus.cc:482] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.123154 15098 raft_consensus.cc:3037] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.131548 14859 tablet_loader.cc:96] loaded metadata for tablet 23db7118b4914afa95eaed713cca8839 (table default.Table_59e7933756ae440cb5377ffc3ca93c29 [id=c68d44dcff8e4c528981ab7206ee4dcb])
I20240506 19:57:03.132905 14859 tablet_loader.cc:96] loaded metadata for tablet 25cb60a953aa486fac4a985ff0fe2441 (table default.Table_014acbfce7444396ad12f76368ca683f [id=767292be27b246929ffa0f7244be6ad7])
I20240506 19:57:03.131548 13249 raft_consensus.cc:348] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.133682 14700 raft_consensus.cc:482] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.133834 14700 raft_consensus.cc:504] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.133414 13249 raft_consensus.cc:374] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.134060 13249 raft_consensus.cc:729] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.134759 14700 leader_election.cc:288] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.134892 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcc036a20824736b16815bcb47645c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.135138 13120 raft_consensus.cc:2454] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:03.135087 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcc036a20824736b16815bcb47645c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.135591 12921 leader_election.cc:341] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.135699 12921 leader_election.cc:302] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.136358 14700 raft_consensus.cc:2781] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.136466 14700 raft_consensus.cc:482] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.136559 14700 raft_consensus.cc:3037] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.135782 13249 consensus_queue.cc:260] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.139256 15098 raft_consensus.cc:504] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.141327 13249 ts_tablet_manager.cc:1398] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240506 19:57:03.141844 13244 tablet_bootstrap.cc:654] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.141958 14700 raft_consensus.cc:504] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.142742 13249 tablet_bootstrap.cc:492] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.146649 14859 tablet_loader.cc:96] loaded metadata for tablet 26f65b90e2394e619eb806a0dbd96bd7 (table default.Table_5eb85e351b7645f5925a458693f85835 [id=bb93eef6ddeb4f759f3db72b85d798b5])
I20240506 19:57:03.147792 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9944e5bfe0a741d69b8f9b6a9dad84dd" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:03.148005 12985 raft_consensus.cc:3037] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.148221 15098 leader_election.cc:288] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.151103 12985 raft_consensus.cc:2454] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:03.151427 13249 tablet_bootstrap.cc:654] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.151598 13057 leader_election.cc:302] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:03.153033 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9944e5bfe0a741d69b8f9b6a9dad84dd" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:03.153645 13058 leader_election.cc:341] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.156242 13245 tablet_bootstrap.cc:492] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.156438 13245 ts_tablet_manager.cc:1367] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.002s
I20240506 19:57:03.156889 15142 raft_consensus.cc:482] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.157184 14859 tablet_loader.cc:96] loaded metadata for tablet 28194038348545dbbae94e20fed2e092 (table default.Table_95c37af6ef334405b443163c4eb60668 [id=2eebc48dfe464f9ba15aec649e8f9d08])
I20240506 19:57:03.157315 14910 raft_consensus.cc:482] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.157275 13245 raft_consensus.cc:348] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.157469 13245 raft_consensus.cc:374] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.157516 13245 raft_consensus.cc:729] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.157431 14910 raft_consensus.cc:504] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.157683 15098 raft_consensus.cc:2781] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.157702 13245 consensus_queue.cc:260] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.158041 14910 leader_election.cc:288] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:03.158617 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.158823 14910 raft_consensus.cc:482] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.158934 14910 raft_consensus.cc:504] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.159118 13245 ts_tablet_manager.cc:1398] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:57:03.159490 14910 leader_election.cc:288] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.159525 13245 tablet_bootstrap.cc:492] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.160096 15098 raft_consensus.cc:686] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:03.160356 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.161957 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df160fd5d8ce4bc192913ce702f8f478" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.157052 15142 raft_consensus.cc:504] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.162427 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6fa6add23b8403ba5a0a65ac6752a5b" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.162523 15142 leader_election.cc:288] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.162865 12781 leader_election.cc:302] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.163300 15142 raft_consensus.cc:2726] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:03.163333 12783 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:57:03.164104 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcc036a20824736b16815bcb47645c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:03.164288 13120 raft_consensus.cc:3037] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.164491 12781 leader_election.cc:302] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.164971 15142 raft_consensus.cc:2726] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:03.165339 12781 leader_election.cc:341] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 60d1f01c0edb4f6b8a621c1643ca1afb
I20240506 19:57:03.165443 12781 leader_election.cc:302] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.165638 15142 raft_consensus.cc:2726] T 60d1f01c0edb4f6b8a621c1643ca1afb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.167304 13120 raft_consensus.cc:2454] T 4dcc036a20824736b16815bcb47645c4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:03.167739 12922 leader_election.cc:302] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:03.168056 15009 raft_consensus.cc:2781] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.168249 15009 raft_consensus.cc:686] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:03.168529 15009 consensus_queue.cc:237] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.169910 14700 leader_election.cc:288] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.170531 13245 tablet_bootstrap.cc:654] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.160785 15098 consensus_queue.cc:237] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.171759 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcc036a20824736b16815bcb47645c4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.171952 14859 tablet_loader.cc:96] loaded metadata for tablet 28b1e772db304a038a46d428b8c522c4 (table default.Table_112c674a2f6a46cb8e60669eb015488f [id=d50032a383514ade85a7e639245332b6])
W20240506 19:57:03.173280 12921 leader_election.cc:341] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.178272 13249 tablet_bootstrap.cc:492] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.178583 13249 ts_tablet_manager.cc:1367] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.003s
I20240506 19:57:03.179311 14859 tablet_loader.cc:96] loaded metadata for tablet 2ab55564dcee4e6196671ac17d3b750c (table default.Table_c649504434cb4d69ac0d74733ff3591e [id=a3c712f1c0db496e908d90ae0af131ee])
I20240506 19:57:03.180255 14859 tablet_loader.cc:96] loaded metadata for tablet 2b949c48806d45c3a74efbc8503c0bdb (table default.Table_9bca56a380f341c28cf5c3f683024098 [id=798e81e44a5a46ec962f48302ab404a8])
I20240506 19:57:03.180200 13249 raft_consensus.cc:348] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.180703 13249 raft_consensus.cc:374] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.180956 13249 raft_consensus.cc:729] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.181066 14859 tablet_loader.cc:96] loaded metadata for tablet 2b975fdb35324fcaa11e5f573fc68b41 (table default.Table_fb159d58ea054e2883302a7bc6ea228b [id=7bdb2c56b7df4c8f89d4af97bc7b402b])
I20240506 19:57:03.181499 13249 consensus_queue.cc:260] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.182454 14859 tablet_loader.cc:96] loaded metadata for tablet 2d407307e422457eab7e4ada0f909024 (table default.Table_ef810fa2e21b4d6ca63fd662ff62cd1e [id=330c88093b4b4bcfa831c77ce9b44115])
I20240506 19:57:03.183223 13249 ts_tablet_manager.cc:1398] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240506 19:57:03.184211 13249 tablet_bootstrap.cc:492] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.185012 14859 tablet_loader.cc:96] loaded metadata for tablet 2db006c642ce4db78c5b7355300e9567 (table default.Table_659a4d19b7d447438d293f59be74f51a [id=a882030e617646b886caac521fb79161])
I20240506 19:57:03.188151 13249 tablet_bootstrap.cc:654] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.190469 13244 tablet_bootstrap.cc:492] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:03.190644 13244 ts_tablet_manager.cc:1367] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.000s
I20240506 19:57:03.191692 13245 tablet_bootstrap.cc:492] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.191634 13244 raft_consensus.cc:348] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.191928 13244 raft_consensus.cc:374] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.191943 13245 ts_tablet_manager.cc:1367] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.007s
I20240506 19:57:03.192023 13244 raft_consensus.cc:729] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.192294 13244 consensus_queue.cc:260] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.192904 13245 raft_consensus.cc:348] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.193228 13245 raft_consensus.cc:374] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.193351 13245 raft_consensus.cc:729] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.193675 13245 consensus_queue.cc:260] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.194474 13245 ts_tablet_manager.cc:1398] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:57:03.194844 13245 tablet_bootstrap.cc:492] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.195930 14915 raft_consensus.cc:482] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.196056 14915 raft_consensus.cc:504] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.196707 14915 leader_election.cc:288] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.197057 15142 raft_consensus.cc:482] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.197186 15142 raft_consensus.cc:504] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.197782 15142 leader_election.cc:288] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.199002 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.199455 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df04fb90230d4bf4a060db993adafd14" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.200251 12781 leader_election.cc:302] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.200528 15142 raft_consensus.cc:2726] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.200945 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "597a48f6950844299f5f292e6a101cae" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.201411 13058 leader_election.cc:341] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.200639 13245 tablet_bootstrap.cc:654] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.210152 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "597a48f6950844299f5f292e6a101cae" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.210405 12985 raft_consensus.cc:2454] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.210755 13057 leader_election.cc:302] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.211004 15104 raft_consensus.cc:2781] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.211082 15104 raft_consensus.cc:482] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.211205 15104 raft_consensus.cc:3037] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.213783 15104 raft_consensus.cc:504] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.213907 13249 tablet_bootstrap.cc:492] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.214427 15104 leader_election.cc:288] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.214582 13249 ts_tablet_manager.cc:1367] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.001s
I20240506 19:57:03.215688 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "597a48f6950844299f5f292e6a101cae" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.215907 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "597a48f6950844299f5f292e6a101cae" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:03.216078 12985 raft_consensus.cc:3037] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.216281 13058 leader_election.cc:341] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.217679 13249 raft_consensus.cc:348] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.218217 13249 raft_consensus.cc:374] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.218434 13249 raft_consensus.cc:729] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.218689 12985 raft_consensus.cc:2454] T 597a48f6950844299f5f292e6a101cae P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:03.219064 13057 leader_election.cc:302] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.219065 13249 consensus_queue.cc:260] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.219784 13244 ts_tablet_manager.cc:1398] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240506 19:57:03.220024 15104 raft_consensus.cc:2781] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.220155 15104 raft_consensus.cc:686] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:03.220297 13249 ts_tablet_manager.cc:1398] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240506 19:57:03.220422 15104 consensus_queue.cc:237] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.221200 13249 tablet_bootstrap.cc:492] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.222869 13244 tablet_bootstrap.cc:492] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:03.224625 13244 tablet_bootstrap.cc:654] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.225361 13249 tablet_bootstrap.cc:654] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.239008 14915 raft_consensus.cc:482] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.239228 14915 raft_consensus.cc:504] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.239961 14915 leader_election.cc:288] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.240294 13245 tablet_bootstrap.cc:492] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.240468 13245 ts_tablet_manager.cc:1367] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.005s
I20240506 19:57:03.241277 13245 raft_consensus.cc:348] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.241492 13245 raft_consensus.cc:374] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.241559 13245 raft_consensus.cc:729] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.242578 13245 consensus_queue.cc:260] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.242806 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4e628b9364cccb525b8e1b8070286" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:57:03.243288 13057 leader_election.cc:341] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.243643 13245 ts_tablet_manager.cc:1398] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:57:03.242555 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4e628b9364cccb525b8e1b8070286" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.244091 13245 tablet_bootstrap.cc:492] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:03.244611 13058 leader_election.cc:341] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.244741 13058 leader_election.cc:302] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.245234 14915 raft_consensus.cc:2726] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.246343 13245 tablet_bootstrap.cc:654] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.248736 13249 tablet_bootstrap.cc:492] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.249029 13249 ts_tablet_manager.cc:1367] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.002s
I20240506 19:57:03.250649 13249 raft_consensus.cc:348] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.251075 13249 raft_consensus.cc:374] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.251214 13249 raft_consensus.cc:729] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.251652 13249 consensus_queue.cc:260] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.257212 13249 ts_tablet_manager.cc:1398] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240506 19:57:03.260344 15141 raft_consensus.cc:482] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.260502 15141 raft_consensus.cc:504] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.261158 15141 leader_election.cc:288] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:03.262384 12781 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 997ecba7fbae41178a25bf7af4d152e1
W20240506 19:57:03.262713 12783 leader_election.cc:341] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 997ecba7fbae41178a25bf7af4d152e1
I20240506 19:57:03.262823 12783 leader_election.cc:302] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.263064 15141 raft_consensus.cc:2726] T 997ecba7fbae41178a25bf7af4d152e1 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.268505 12843 raft_consensus.cc:3037] T e63baf59dc8244daad831b2987a51cfb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.275324 13245 tablet_bootstrap.cc:492] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.275833 13245 ts_tablet_manager.cc:1367] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.007s
I20240506 19:57:03.277973 13245 raft_consensus.cc:348] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.278520 13245 raft_consensus.cc:374] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.278762 13245 raft_consensus.cc:729] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.280160 13244 tablet_bootstrap.cc:492] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:03.280339 13244 ts_tablet_manager.cc:1367] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20240506 19:57:03.279551 13245 consensus_queue.cc:260] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.281200 13244 raft_consensus.cc:348] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.281474 13244 raft_consensus.cc:374] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.281567 13244 raft_consensus.cc:729] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.281371 13249 tablet_bootstrap.cc:492] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.283306 15141 raft_consensus.cc:482] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.283425 15141 raft_consensus.cc:504] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.283629 13245 ts_tablet_manager.cc:1398] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240506 19:57:03.284029 15141 leader_election.cc:288] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:03.284857 12783 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 452a83470585433e8def7174300d3a7f
I20240506 19:57:03.285351 13245 tablet_bootstrap.cc:492] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:03.286819 12781 leader_election.cc:341] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 452a83470585433e8def7174300d3a7f
I20240506 19:57:03.286952 12781 leader_election.cc:302] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.287369 15141 raft_consensus.cc:2726] T 452a83470585433e8def7174300d3a7f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.291261 14859 tablet_loader.cc:96] loaded metadata for tablet 2f45e9fa2cb945d2a04fa716d1ab33b4 (table default.Table_5de7f6e4f9f5452291f85ec5db486f81 [id=ea7a610430e34ca2b3ab4868800722d5])
I20240506 19:57:03.294400 13249 tablet_bootstrap.cc:654] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.298386 13245 tablet_bootstrap.cc:654] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.299826 13244 consensus_queue.cc:260] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.300704 15104 consensus_queue.cc:1035] T 1b89fb24d94042e1bfa772265a991680 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.306002 13244 ts_tablet_manager.cc:1398] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240506 19:57:03.306375 13244 tablet_bootstrap.cc:492] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:03.311677 14700 consensus_queue.cc:1035] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.319620 14859 tablet_loader.cc:96] loaded metadata for tablet 30beced0d19f4c11bf5bd0a5d3c05af7 (table default.Table_1f73021793a44d038dfb378928d261d3 [id=c7ee5ba59ccf47c39ebbce23f5663c3d])
I20240506 19:57:03.323050 15098 raft_consensus.cc:482] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.323285 15098 raft_consensus.cc:504] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.324196 15098 leader_election.cc:288] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.324959 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35453f75a3494f8baf13e0d5cacc9003" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.325197 12986 raft_consensus.cc:2454] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.325628 13057 leader_election.cc:302] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:03.325949 15098 raft_consensus.cc:2781] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.325363 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35453f75a3494f8baf13e0d5cacc9003" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.326081 15098 raft_consensus.cc:482] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.326259 15098 raft_consensus.cc:3037] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.327244 13058 leader_election.cc:341] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.329654 14859 tablet_loader.cc:96] loaded metadata for tablet 3141b13136c147b8b9d199c9728208e5 (table default.Table_9caff027d79b4a8cbdde4ba6074c1e90 [id=3f77354fa69f4941abc4db4fd5b70b7a])
I20240506 19:57:03.330988 12843 raft_consensus.cc:3037] T 6089f4f68bfe415e8d97dfd6849b7714 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.335191 14859 tablet_loader.cc:96] loaded metadata for tablet 32fec440f0cc4490a71eb37d6a1155ae (table default.Table_11ace0d42c9840d19108e3bfa5f1bb71 [id=29d018e8e77c4cbfa4cec78985b4de98])
I20240506 19:57:03.335594 15098 raft_consensus.cc:504] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.338284 15104 raft_consensus.cc:482] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.339484 15104 raft_consensus.cc:504] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.339702 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35453f75a3494f8baf13e0d5cacc9003" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:03.339913 12985 raft_consensus.cc:3037] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.340421 15104 leader_election.cc:288] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.340548 15098 leader_election.cc:288] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.340548 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d37bb1c2144c4fa71aab6113b9f06e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:57:03.341051 13057 leader_election.cc:341] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.343405 12985 raft_consensus.cc:2454] T 35453f75a3494f8baf13e0d5cacc9003 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:03.343809 13057 leader_election.cc:302] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:03.344099 15098 raft_consensus.cc:2781] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.344218 15098 raft_consensus.cc:686] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:03.336470 14859 tablet_loader.cc:96] loaded metadata for tablet 33a082ee0898444ba9f0f5ee92ba89cc (table default.Table_946004ac61bd4421abf9cdfa64f35460 [id=1881dd13d96a4563a72c4791dcaccfca])
I20240506 19:57:03.346522 14859 tablet_loader.cc:96] loaded metadata for tablet 3411f2ae4a4e4e1b8609292f7ac86c70 (table default.Table_cd4a27060d284b228e7949d9118b32ad [id=fe6cdb333fb141cba8e81172bdb7594b])
I20240506 19:57:03.348994 15098 consensus_queue.cc:237] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.349807 14859 tablet_loader.cc:96] loaded metadata for tablet 3450e9b8544d4accb79173d5166ded49 (table default.Table_0821a294cda34e98b1de703d3b5f8c1a [id=4d7b052cb45e4c1e9b33fdb0648f12d8])
I20240506 19:57:03.351377 14859 tablet_loader.cc:96] loaded metadata for tablet 35453f75a3494f8baf13e0d5cacc9003 (table default.Table_0778153fb8284502a1f3e86332825c48 [id=a33185b46fbc424e9e81cef309eb2cdb])
I20240506 19:57:03.355240 14859 tablet_loader.cc:96] loaded metadata for tablet 371d450876ff40a59b859f72f4e20c5a (table default.Table_6f20f6b912c249479dbf461093583817 [id=d401826620ea46ad8df6016c1f1cf968])
I20240506 19:57:03.357862 14859 tablet_loader.cc:96] loaded metadata for tablet 37353519fb1343349193ea6c82ecda89 (table default.Table_eea6723475c7480f86f5246eb849bf13 [id=62405a21681d4effaea8f463eb15bf4d])
I20240506 19:57:03.359079 12847 raft_consensus.cc:3037] T bc85233b180f4b30a86bb65c86af684e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.375469 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.193:39475, user_credentials={real_user=slave}} blocked reactor thread for 48284us
I20240506 19:57:03.377351 13249 tablet_bootstrap.cc:492] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.377614 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d37bb1c2144c4fa71aab6113b9f06e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.377749 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35453f75a3494f8baf13e0d5cacc9003" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.377974 15098 raft_consensus.cc:482] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.378105 15098 raft_consensus.cc:504] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.378759 15098 leader_election.cc:288] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.378896 13249 ts_tablet_manager.cc:1367] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.008s
W20240506 19:57:03.381589 12921 consensus_peers.cc:485] T ea7e073296ce46e786362aaed96eb4dd P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:03.381942 13058 leader_election.cc:341] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.382052 13058 leader_election.cc:302] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.382264 15098 raft_consensus.cc:2726] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.382266 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3291123a1e42af80dbfb578c855618" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:57:03.382725 13058 leader_election.cc:341] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:03.382934 13057 leader_election.cc:341] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.383109 15142 raft_consensus.cc:482] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.383255 15142 raft_consensus.cc:504] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.383092 13249 raft_consensus.cc:348] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.383882 15142 leader_election.cc:288] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.383963 13249 raft_consensus.cc:374] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.384444 13249 raft_consensus.cc:729] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.385807 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3291123a1e42af80dbfb578c855618" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.386152 12842 raft_consensus.cc:3037] T df04fb90230d4bf4a060db993adafd14 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.386752 13058 leader_election.cc:341] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.386855 13058 leader_election.cc:302] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.385370 13249 consensus_queue.cc:260] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.387064 15098 raft_consensus.cc:2726] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:03.385823 12783 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d7be9788df8e47d285bd932d078c6a8b
W20240506 19:57:03.390846 12781 leader_election.cc:341] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d7be9788df8e47d285bd932d078c6a8b
I20240506 19:57:03.390967 12781 leader_election.cc:302] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.391376 15142 raft_consensus.cc:2726] T d7be9788df8e47d285bd932d078c6a8b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.391359 13249 ts_tablet_manager.cc:1398] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240506 19:57:03.397186 13249 tablet_bootstrap.cc:492] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.403985 13245 tablet_bootstrap.cc:492] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.404353 13245 ts_tablet_manager.cc:1367] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.010s
I20240506 19:57:03.406056 13245 raft_consensus.cc:348] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.406560 13245 raft_consensus.cc:374] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.406721 13245 raft_consensus.cc:729] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.407159 13245 consensus_queue.cc:260] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.408354 13245 ts_tablet_manager.cc:1398] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:03.409269 13245 tablet_bootstrap.cc:492] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.414038 13249 tablet_bootstrap.cc:654] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.415210 13245 tablet_bootstrap.cc:654] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.417609 12843 raft_consensus.cc:3037] T b6fa6add23b8403ba5a0a65ac6752a5b P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.423074 14656 raft_consensus.cc:482] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 19:57:03.423096 12921 consensus_peers.cc:485] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.423313 14656 raft_consensus.cc:504] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.424242 14656 leader_election.cc:288] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.424371 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c88f80f5b34672b7c3a4b604469bf1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.424630 12985 raft_consensus.cc:2454] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.424835 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c88f80f5b34672b7c3a4b604469bf1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.425356 13058 leader_election.cc:341] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.425839 13057 leader_election.cc:302] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.426311 14656 raft_consensus.cc:2781] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.426435 14656 raft_consensus.cc:482] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.426559 14656 raft_consensus.cc:3037] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.427134 12921 consensus_peers.cc:485] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.429556 14656 raft_consensus.cc:504] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.430344 14656 leader_election.cc:288] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.430872 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c88f80f5b34672b7c3a4b604469bf1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:03.433835 13058 leader_election.cc:341] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.434311 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c88f80f5b34672b7c3a4b604469bf1" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:03.434749 12985 raft_consensus.cc:3037] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.438242 13249 tablet_bootstrap.cc:492] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.438514 13249 ts_tablet_manager.cc:1367] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.004s
I20240506 19:57:03.440191 13249 raft_consensus.cc:348] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.440555 13249 raft_consensus.cc:374] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.440685 13249 raft_consensus.cc:729] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.441093 13249 consensus_queue.cc:260] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.448683 13245 tablet_bootstrap.cc:492] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.448995 13245 ts_tablet_manager.cc:1367] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.009s
I20240506 19:57:03.451035 12848 raft_consensus.cc:3037] T 8d09eafa82a0423a8358958e78f728e2 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.450861 13245 raft_consensus.cc:348] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.451259 13245 raft_consensus.cc:374] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.451395 13245 raft_consensus.cc:729] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.451819 13245 consensus_queue.cc:260] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.453102 13249 ts_tablet_manager.cc:1398] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240506 19:57:03.453902 13245 ts_tablet_manager.cc:1398] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:57:03.455369 13245 tablet_bootstrap.cc:492] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.455667 12985 raft_consensus.cc:2454] T 58c88f80f5b34672b7c3a4b604469bf1 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:03.456063 13057 leader_election.cc:302] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.458539 13245 tablet_bootstrap.cc:654] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.460392 14656 raft_consensus.cc:2781] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.460503 14656 raft_consensus.cc:686] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:03.460755 14656 consensus_queue.cc:237] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.461030 13249 tablet_bootstrap.cc:492] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.464363 13249 tablet_bootstrap.cc:654] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.472389 12845 raft_consensus.cc:3037] T 64485c3c13e94055a019f2ef7be1f2fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.477631 14915 raft_consensus.cc:482] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.477790 14915 raft_consensus.cc:504] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.478358 14915 leader_election.cc:288] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.479395 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b899244c35b4bbe9e916c778111f0c0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.479859 13058 leader_election.cc:341] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.479904 12846 raft_consensus.cc:3037] T 11b00381c6aa42c9b969410b51ebe81e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:03.492048 13058 consensus_peers.cc:485] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.503813 15011 consensus_queue.cc:1035] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.509637 13249 tablet_bootstrap.cc:492] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.510939 13249 ts_tablet_manager.cc:1367] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.003s
I20240506 19:57:03.520891 15009 consensus_queue.cc:1035] T d0aae034f52645ba96827923f4fa305c P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.519233 13249 raft_consensus.cc:348] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.521442 13249 raft_consensus.cc:374] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.522080 13249 raft_consensus.cc:729] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.522835 12844 raft_consensus.cc:3037] T df160fd5d8ce4bc192913ce702f8f478 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.523635 13249 consensus_queue.cc:260] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.527853 13249 ts_tablet_manager.cc:1398] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.016s user 0.003s sys 0.001s
I20240506 19:57:03.530592 13249 tablet_bootstrap.cc:492] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.531556 14915 raft_consensus.cc:482] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.531701 14915 raft_consensus.cc:504] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.532409 14915 leader_election.cc:288] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.533572 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216f98cb6ac4e249ca2653518ff9e87" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:03.533784 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216f98cb6ac4e249ca2653518ff9e87" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.533975 12986 raft_consensus.cc:2454] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
W20240506 19:57:03.534466 13058 leader_election.cc:341] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.534971 13057 leader_election.cc:302] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.535589 14915 raft_consensus.cc:2781] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.535687 14915 raft_consensus.cc:482] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.535765 14915 raft_consensus.cc:3037] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.537974 13249 tablet_bootstrap.cc:654] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.539602 14915 raft_consensus.cc:504] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.541348 14915 leader_election.cc:288] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.541313 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216f98cb6ac4e249ca2653518ff9e87" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:03.541666 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216f98cb6ac4e249ca2653518ff9e87" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:03.541875 12986 raft_consensus.cc:3037] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.545495 12986 raft_consensus.cc:2454] T 4216f98cb6ac4e249ca2653518ff9e87 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:03.545951 15142 raft_consensus.cc:482] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.546162 15142 raft_consensus.cc:504] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.547037 15142 leader_election.cc:288] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.549108 15142 raft_consensus.cc:482] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.549276 15142 raft_consensus.cc:504] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.550063 15142 leader_election.cc:288] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:03.551113 12781 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 08dafb661b2443f5afadc6816868bac8
W20240506 19:57:03.552680 12783 leader_election.cc:341] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 08dafb661b2443f5afadc6816868bac8
I20240506 19:57:03.552829 12783 leader_election.cc:302] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.553514 15142 raft_consensus.cc:2726] T 08dafb661b2443f5afadc6816868bac8 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.553902 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20016b56be94248905e0e8d6a35100f" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.543337 13245 tablet_bootstrap.cc:492] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
W20240506 19:57:03.547655 13058 leader_election.cc:341] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.556463 13245 ts_tablet_manager.cc:1367] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.101s user 0.007s sys 0.006s
I20240506 19:57:03.556617 13057 leader_election.cc:302] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.557163 15104 raft_consensus.cc:2781] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:03.552332 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20016b56be94248905e0e8d6a35100f" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:03.557276 15104 raft_consensus.cc:686] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:03.557184 13245 raft_consensus.cc:348] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.557730 13245 raft_consensus.cc:374] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.557788 13245 raft_consensus.cc:729] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.557737 15104 consensus_queue.cc:237] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.557966 13245 consensus_queue.cc:260] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.558432 13245 ts_tablet_manager.cc:1398] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:03.558686 13245 tablet_bootstrap.cc:492] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.559903 15009 consensus_queue.cc:1035] T ffbb87878d06417cbabf3c218728dbdc P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.560225 13245 tablet_bootstrap.cc:654] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.561108 15009 consensus_queue.cc:1035] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.561534 13249 tablet_bootstrap.cc:492] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.561810 13249 ts_tablet_manager.cc:1367] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.006s
I20240506 19:57:03.565582 12783 leader_election.cc:302] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:03.566260 15142 raft_consensus.cc:2726] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:03.570132 12921 consensus_peers.cc:485] T 08535fa1600b41ff919fefa93cfc4c78 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:03.574101 12921 consensus_peers.cc:485] T f078559df48f4d23869ddcb4b3fca9fc P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.575284 15142 consensus_queue.cc:1035] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.576694 13249 raft_consensus.cc:348] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.578836 13249 raft_consensus.cc:374] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.579334 13249 raft_consensus.cc:729] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.581698 13249 consensus_queue.cc:260] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.587260 13249 ts_tablet_manager.cc:1398] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.025s user 0.002s sys 0.002s
I20240506 19:57:03.591293 13249 tablet_bootstrap.cc:492] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.596056 14915 raft_consensus.cc:482] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.597409 14915 raft_consensus.cc:504] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.598961 13245 tablet_bootstrap.cc:492] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.599679 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eac98ac8f040c58f61106bb0e989de" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
W20240506 19:57:03.600200 13057 leader_election.cc:341] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.600097 13245 ts_tablet_manager.cc:1367] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.010s
I20240506 19:57:03.604203 13245 raft_consensus.cc:348] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.605134 13249 tablet_bootstrap.cc:654] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.605196 13245 raft_consensus.cc:374] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.605490 13245 raft_consensus.cc:729] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.606495 13245 consensus_queue.cc:260] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.607857 13245 ts_tablet_manager.cc:1398] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240506 19:57:03.608417 13245 tablet_bootstrap.cc:492] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.613216 13245 tablet_bootstrap.cc:654] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.615813 14915 leader_election.cc:288] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.616515 12841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eac98ac8f040c58f61106bb0e989de" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.617134 13058 leader_election.cc:341] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.617621 13058 leader_election.cc:302] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.623247 14915 raft_consensus.cc:2726] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:03.627012 13249 tablet_bootstrap.cc:492] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.627169 13249 ts_tablet_manager.cc:1367] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20240506 19:57:03.628013 13249 raft_consensus.cc:348] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.628237 13249 raft_consensus.cc:374] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.628227 13245 tablet_bootstrap.cc:492] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.628304 13249 raft_consensus.cc:729] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.628504 13249 consensus_queue.cc:260] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.628630 13245 ts_tablet_manager.cc:1367] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240506 19:57:03.629043 13249 ts_tablet_manager.cc:1398] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:03.629354 13249 tablet_bootstrap.cc:492] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.630937 13249 tablet_bootstrap.cc:654] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.632066 13245 raft_consensus.cc:348] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.632544 13245 raft_consensus.cc:374] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.632731 13245 raft_consensus.cc:729] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.633122 13245 consensus_queue.cc:260] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.634215 13245 ts_tablet_manager.cc:1398] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:57:03.634773 13245 tablet_bootstrap.cc:492] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.639520 13245 tablet_bootstrap.cc:654] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.639603 12841 raft_consensus.cc:3037] T a20016b56be94248905e0e8d6a35100f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.646668 13249 tablet_bootstrap.cc:492] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.646826 13249 ts_tablet_manager.cc:1367] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240506 19:57:03.647687 13249 raft_consensus.cc:348] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.647887 13249 raft_consensus.cc:374] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.647943 13249 raft_consensus.cc:729] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.648123 13249 consensus_queue.cc:260] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.648698 13249 ts_tablet_manager.cc:1398] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:03.648958 13249 tablet_bootstrap.cc:492] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.655566 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b899244c35b4bbe9e916c778111f0c0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.655900 12985 raft_consensus.cc:2454] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.657377 13057 leader_election.cc:302] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.660387 14915 raft_consensus.cc:2781] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.661351 14915 raft_consensus.cc:482] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.661507 14915 raft_consensus.cc:3037] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.676648 14859 tablet_loader.cc:96] loaded metadata for tablet 3a37ab9bc9784eb6b1b0f2f1ccc7c5ef (table default.Table_c0a9791431af47eaafcd42d67421b82e [id=5a2a63588e864676bbbaf4ab586a819d])
I20240506 19:57:03.681947 13244 tablet_bootstrap.cc:654] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:03.687932 12783 consensus_peers.cc:485] T f3a8531a1793404699532295164a3895 P ef18e88b87064ed4b6c20f9325ebcc33 -> Peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Couldn't send request to peer 8a9c37db9d4f41cd93820a1c81f48d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3a8531a1793404699532295164a3895. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.694819 13249 tablet_bootstrap.cc:654] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.701690 15103 consensus_queue.cc:1035] T 744864043f9347e0a13f8c2a7d6b147f P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.703837 15103 raft_consensus.cc:482] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.704020 15103 raft_consensus.cc:504] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.704754 15103 leader_election.cc:288] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:03.706174 12921 consensus_peers.cc:485] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.708020 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc22261ab2034e818834ba97c2760dad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.708716 13058 leader_election.cc:341] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.723883 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc22261ab2034e818834ba97c2760dad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.724160 12987 raft_consensus.cc:2454] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.741645 15228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27d4482cd59a4e5687992b1f8a177373)
I20240506 19:57:03.742076 15228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a2eabb30d4154960a44eb50a8a87d" member_type: VOTER last_known_addr { host: "127.12.48.254" port: 33495 } } peers { permanent_uuid: "27d4482cd59a4e5687992b1f8a177373" member_type: VOTER last_known_addr { host: "127.12.48.253" port: 36819 } } peers { permanent_uuid: "97f390a786034cddbff757779037f861" member_type: VOTER last_known_addr { host: "127.12.48.252" port: 36867 } }
I20240506 19:57:03.743901 15228 leader_election.cc:288] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495), 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819)
I20240506 19:57:03.754140 15142 raft_consensus.cc:482] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.755057 15142 raft_consensus.cc:504] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.758353 15142 leader_election.cc:288] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.761416 14859 tablet_loader.cc:96] loaded metadata for tablet 3b74c481671149a6bc6b138b8b2d7e42 (table default.Table_5b7eae44ba9746f9b088affbf1a88cdb [id=273f7fc51349495f90765ff0b2ba2c2f])
I20240506 19:57:03.762529 14859 tablet_loader.cc:96] loaded metadata for tablet 3bb5a12521044102ae55ca953f1496f9 (table default.Table_085ebc849e674a89a8b03e33327a342d [id=ba90b3aba0c346559318504ce789c81e])
I20240506 19:57:03.763513 14859 tablet_loader.cc:96] loaded metadata for tablet 3c882759dc0f4ba58aafaab3b855fa30 (table default.Table_1d4371bd92c34570ba9ffca5ae2a387d [id=224d462b1c6d4535937a0d82ff847771])
I20240506 19:57:03.764552 14859 tablet_loader.cc:96] loaded metadata for tablet 3f5bb166d2e740f59be13faa6e92cdb3 (table default.Table_7a1c1e363dbc41dc9708971c2656dfe5 [id=3a6f58959205421cb5894d9424959ea0])
I20240506 19:57:03.773166 15103 raft_consensus.cc:482] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.773357 15103 raft_consensus.cc:504] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.774123 15103 leader_election.cc:288] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:03.775440 12783 leader_election.cc:341] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 673dc7c83f8748a09c41d51b42214aaf
I20240506 19:57:03.776161 15142 raft_consensus.cc:482] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.776819 15142 raft_consensus.cc:504] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:03.780237 12922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.48.195:40535, user_credentials={real_user=slave}} blocked reactor thread for 56449us
W20240506 19:57:03.780782 12783 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: b2be78cbc8264f6e8d2e05137135161c
I20240506 19:57:03.780867 14700 consensus_queue.cc:1035] T 1e22e5a015f74526b49363bc789cb4ed P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:03.783270 15142 leader_election.cc:288] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.802687 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.803548 13058 leader_election.cc:341] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.851857 13057 leader_election.cc:302] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:03.853003 15098 raft_consensus.cc:2781] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:03.853122 15098 raft_consensus.cc:482] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:03.853200 15098 raft_consensus.cc:3037] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:03.858187 15142 raft_consensus.cc:482] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.858397 15142 raft_consensus.cc:504] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.859129 15142 leader_election.cc:288] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:03.860473 12783 leader_election.cc:341] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 04738def56324a368f8182543107e32e
I20240506 19:57:03.868136 15098 raft_consensus.cc:504] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:03.873927 14797 raft_consensus.cc:482] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.874116 14797 raft_consensus.cc:504] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.870303 14656 raft_consensus.cc:482] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.874569 14656 raft_consensus.cc:504] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.875178 14797 leader_election.cc:288] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.876276 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee69d8e09c2f4c898ce547ec396ca0e6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.876797 12921 leader_election.cc:341] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.877892 14656 leader_election.cc:288] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.879513 14915 raft_consensus.cc:504] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.880342 14915 leader_election.cc:288] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.881947 15098 leader_election.cc:288] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.892676 13249 tablet_bootstrap.cc:492] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.892824 13249 ts_tablet_manager.cc:1367] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.244s user 0.011s sys 0.019s
I20240506 19:57:03.893616 13249 raft_consensus.cc:348] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.893810 13249 raft_consensus.cc:374] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.893873 13249 raft_consensus.cc:729] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.894079 13249 consensus_queue.cc:260] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.894659 13249 ts_tablet_manager.cc:1398] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:03.894923 13249 tablet_bootstrap.cc:492] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:03.896579 13249 tablet_bootstrap.cc:654] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.895164 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b899244c35b4bbe9e916c778111f0c0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:03.899408 13058 leader_election.cc:341] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.900059 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc22261ab2034e818834ba97c2760dad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:03.900710 13058 leader_election.cc:341] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.900779 15142 raft_consensus.cc:482] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.900907 15142 raft_consensus.cc:504] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.901893 15142 leader_election.cc:288] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:03.902628 12783 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:57:03.905480 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f3def738fb4cb195d08e48d7d2b293" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.909619 13058 leader_election.cc:341] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:03.910190 13244 tablet_bootstrap.cc:492] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:03.910475 13244 ts_tablet_manager.cc:1367] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.604s user 0.015s sys 0.001s
I20240506 19:57:03.911542 13244 raft_consensus.cc:348] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.911914 13244 raft_consensus.cc:374] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.912101 13244 raft_consensus.cc:729] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.912667 13244 consensus_queue.cc:260] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
W20240506 19:57:03.934449 12921 consensus_peers.cc:485] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.945364 15142 raft_consensus.cc:482] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.945660 15142 raft_consensus.cc:504] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.946637 15142 leader_election.cc:288] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.948181 13244 ts_tablet_manager.cc:1398] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20240506 19:57:03.948503 15248 raft_consensus.cc:482] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.948635 15248 raft_consensus.cc:504] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.949576 13245 tablet_bootstrap.cc:492] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.949792 13245 ts_tablet_manager.cc:1367] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.315s user 0.109s sys 0.177s
I20240506 19:57:03.950814 13245 raft_consensus.cc:348] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.951089 13245 raft_consensus.cc:374] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.951189 13245 raft_consensus.cc:729] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.952010 15248 leader_election.cc:288] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.951946 13245 consensus_queue.cc:260] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.952561 13245 ts_tablet_manager.cc:1398] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:03.953120 13245 tablet_bootstrap.cc:492] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:03.954129 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cade6b1c5a034b23aa33b2349e1dab95" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.955040 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.955221 12987 raft_consensus.cc:2454] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.955425 13245 tablet_bootstrap.cc:654] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.957371 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f3def738fb4cb195d08e48d7d2b293" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:03.957556 12987 raft_consensus.cc:2454] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:03.958057 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:03.959023 12921 leader_election.cc:341] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:03.959717 12921 consensus_peers.cc:485] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:03.960734 14859 tablet_loader.cc:96] loaded metadata for tablet 3f7984f6e0fe485fa0c15fd13c3f428e (table default.Table_2cbe1f98afef411bbef910c6931cabd1 [id=8180a10700c54e20b558796b518d74b4])
I20240506 19:57:03.961308 14859 tablet_loader.cc:96] loaded metadata for tablet 3f7aceb691114827821194eb4b6760fc (table default.Table_7297af847f20409aae62497c2b197328 [id=edf335649ed04b0a9027ba71825918ee])
I20240506 19:57:03.962787 14859 tablet_loader.cc:96] loaded metadata for tablet 41853001e53948a49fbb75b093dda93d (table default.Table_df06dbccf0ce480489e9ff86ef1be62d [id=59d2236266164cda96142253cc0a869a])
I20240506 19:57:03.963193 14859 tablet_loader.cc:96] loaded metadata for tablet 418c79b0361f4b2480d4cfbadeef6dc8 (table default.Table_680d6530958740e293cd6eddcad88b21 [id=919e8b8b265b452e8972fa41f11645a9])
I20240506 19:57:03.963588 14859 tablet_loader.cc:96] loaded metadata for tablet 418e02201bfc4a34a22612c2edc3e05f (table default.Table_d486818ffe5644efa8b0211637048aa3 [id=7e16a8d752884de9a4b54a8f7a376ddf])
I20240506 19:57:03.964762 14859 tablet_loader.cc:96] loaded metadata for tablet 43460a21f28c41fdb23a6d7f2e942837 (table default.Table_86e7dc4a07f542679409dcc2a3187f46 [id=cdb79dcc2a134fb788f6f1e285ac593d])
I20240506 19:57:03.971247 15270 raft_consensus.cc:482] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.971652 15270 raft_consensus.cc:504] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:03.973385 15270 leader_election.cc:288] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:03.978034 13244 tablet_bootstrap.cc:492] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:03.979853 13244 tablet_bootstrap.cc:654] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:03.993168 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cade6b1c5a034b23aa33b2349e1dab95" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:03.995035 12783 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:57:03.995214 13249 tablet_bootstrap.cc:492] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:03.995375 13249 ts_tablet_manager.cc:1367] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.002s
I20240506 19:57:03.996161 13244 tablet_bootstrap.cc:492] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:03.996079 13249 raft_consensus.cc:348] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.996294 13249 raft_consensus.cc:374] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:03.996359 13249 raft_consensus.cc:729] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:03.996624 15098 raft_consensus.cc:482] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:03.996553 13249 consensus_queue.cc:260] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.996748 15098 raft_consensus.cc:504] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:03.997061 13249 ts_tablet_manager.cc:1398] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:03.997324 15098 leader_election.cc:288] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:03.998934 13245 tablet_bootstrap.cc:492] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:03.999293 13245 ts_tablet_manager.cc:1367] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.019s
I20240506 19:57:03.999533 13244 ts_tablet_manager.cc:1367] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s
I20240506 19:57:04.000419 13244 raft_consensus.cc:348] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.000811 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b899244c35b4bbe9e916c778111f0c0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.000998 12987 raft_consensus.cc:3037] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.001019 13244 raft_consensus.cc:374] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.001199 13244 raft_consensus.cc:729] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.001086 13245 raft_consensus.cc:348] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.001534 13245 raft_consensus.cc:374] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.001638 13245 raft_consensus.cc:729] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.001595 13244 consensus_queue.cc:260] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.002049 13245 consensus_queue.cc:260] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.003196 13245 ts_tablet_manager.cc:1398] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:57:04.003700 12987 raft_consensus.cc:2454] T 0b899244c35b4bbe9e916c778111f0c0 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:04.003798 13245 tablet_bootstrap.cc:492] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:04.004509 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc22261ab2034e818834ba97c2760dad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.004673 12987 raft_consensus.cc:3037] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:04.007371 12921 consensus_peers.cc:485] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.007412 12987 raft_consensus.cc:2454] T cc22261ab2034e818834ba97c2760dad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:04.007515 13245 tablet_bootstrap.cc:654] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.009078 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30a2ce53464d12b98c6a0c2fdeb4fa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.009308 12986 raft_consensus.cc:2454] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:04.002486 13244 ts_tablet_manager.cc:1398] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240506 19:57:04.010638 14089 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.48.254:33495, user_credentials={real_user=slave}} blocked reactor thread for 71223.8us
I20240506 19:57:04.011302 13244 tablet_bootstrap.cc:492] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.012993 13244 tablet_bootstrap.cc:654] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:04.013135 14089 leader_election.cc:334] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111)
I20240506 19:57:04.017040 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30a2ce53464d12b98c6a0c2fdeb4fa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.017647 13058 leader_election.cc:341] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.018431 13111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee69d8e09c2f4c898ce547ec396ca0e6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.018631 13111 raft_consensus.cc:2454] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:04.032876 13244 tablet_bootstrap.cc:492] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.033154 13244 ts_tablet_manager.cc:1367] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240506 19:57:04.034045 13244 raft_consensus.cc:348] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.034336 13244 raft_consensus.cc:374] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.034453 13244 raft_consensus.cc:729] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.034713 13244 consensus_queue.cc:260] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.036149 13244 ts_tablet_manager.cc:1398] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:04.036967 13244 tablet_bootstrap.cc:492] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.038808 13244 tablet_bootstrap.cc:654] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.053877 13244 tablet_bootstrap.cc:492] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.054167 13244 ts_tablet_manager.cc:1367] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240506 19:57:04.054925 13244 raft_consensus.cc:348] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.055209 13244 raft_consensus.cc:374] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.055310 13244 raft_consensus.cc:729] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.057857 13244 consensus_queue.cc:260] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.059206 13244 ts_tablet_manager.cc:1398] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:04.059717 13244 tablet_bootstrap.cc:492] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.061594 13244 tablet_bootstrap.cc:654] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.064474 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.064733 13116 raft_consensus.cc:2454] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
W20240506 19:57:04.065398 12781 leader_election.cc:341] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: b2be78cbc8264f6e8d2e05137135161c
I20240506 19:57:04.065560 12781 leader_election.cc:302] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.066270 15268 raft_consensus.cc:2726] T b2be78cbc8264f6e8d2e05137135161c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:04.067437 12781 leader_election.cc:341] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 673dc7c83f8748a09c41d51b42214aaf
I20240506 19:57:04.067588 12781 leader_election.cc:302] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.068167 15268 raft_consensus.cc:2726] T 673dc7c83f8748a09c41d51b42214aaf P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:04.069031 12781 leader_election.cc:341] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 04738def56324a368f8182543107e32e
I20240506 19:57:04.069180 12781 leader_election.cc:302] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.069739 15268 raft_consensus.cc:2726] T 04738def56324a368f8182543107e32e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:04.070236 12781 leader_election.cc:341] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 0aa8f9aaca444c839a88c07f58cb6ad6
I20240506 19:57:04.070380 12781 leader_election.cc:302] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.070873 15268 raft_consensus.cc:2726] T 0aa8f9aaca444c839a88c07f58cb6ad6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.072199 12781 leader_election.cc:302] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.072714 15268 raft_consensus.cc:2726] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.073004 15268 raft_consensus.cc:482] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.073154 15268 raft_consensus.cc:504] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.073833 15268 leader_election.cc:288] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
W20240506 19:57:04.074930 12783 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:57:04.076004 13249 tablet_bootstrap.cc:492] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.079989 13249 tablet_bootstrap.cc:654] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.082645 13244 tablet_bootstrap.cc:492] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.083222 13244 ts_tablet_manager.cc:1367] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20240506 19:57:04.084139 13244 raft_consensus.cc:348] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.085343 13244 raft_consensus.cc:374] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.085613 13244 raft_consensus.cc:729] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.086076 13244 consensus_queue.cc:260] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.087903 13244 ts_tablet_manager.cc:1398] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 19:57:04.091675 14859 tablet_loader.cc:96] loaded metadata for tablet 4531f67f8fb34af5b12ef1c3316f84b9 (table default.Table_e74f7640674041f292cd9a7b4ffb3e9e [id=22080408cdb6434b89a4834e857cf550])
I20240506 19:57:04.092655 14859 tablet_loader.cc:96] loaded metadata for tablet 45421a6499f2468dadf9fb10866b0144 (table default.Table_ddc8aae93ec243fd9f2aa7841c96bbca [id=505e004369be45e3950f15e43d47a14e])
I20240506 19:57:04.100345 13057 leader_election.cc:302] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.101047 14656 raft_consensus.cc:2781] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.101166 14656 raft_consensus.cc:482] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.101243 14656 raft_consensus.cc:3037] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.105104 14656 raft_consensus.cc:504] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.107576 14656 leader_election.cc:288] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.108093 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.108821 13058 leader_election.cc:341] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.109392 13244 tablet_bootstrap.cc:492] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.111196 13244 tablet_bootstrap.cc:654] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.111455 13057 leader_election.cc:302] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.111908 14656 raft_consensus.cc:2781] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.111992 14656 raft_consensus.cc:482] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.112058 14656 raft_consensus.cc:3037] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.114665 14656 raft_consensus.cc:504] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.115409 14656 leader_election.cc:288] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.117928 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f3def738fb4cb195d08e48d7d2b293" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.118552 13058 leader_election.cc:341] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.122972 13244 tablet_bootstrap.cc:492] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.123190 13244 ts_tablet_manager.cc:1367] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240506 19:57:04.123966 13244 raft_consensus.cc:348] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.124298 13244 raft_consensus.cc:374] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.124409 13244 raft_consensus.cc:729] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.124445 13057 leader_election.cc:302] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.124720 13244 consensus_queue.cc:260] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.125263 15098 raft_consensus.cc:2781] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.125419 15098 raft_consensus.cc:686] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.125526 13244 ts_tablet_manager.cc:1398] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:04.125833 13244 tablet_bootstrap.cc:492] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.125681 15098 consensus_queue.cc:237] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.127898 13244 tablet_bootstrap.cc:654] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.139772 13057 leader_election.cc:302] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.140384 14915 raft_consensus.cc:2781] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.140523 14915 raft_consensus.cc:686] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.140826 14915 consensus_queue.cc:237] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.144707 15268 raft_consensus.cc:482] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.144892 15268 raft_consensus.cc:504] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.145572 15268 leader_election.cc:288] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:04.147176 13058 consensus_peers.cc:485] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:04.151451 12783 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:57:04.174302 12847 raft_consensus.cc:3037] T 986f9c83b04f41b49e9b78fce731a90f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.181102 14656 raft_consensus.cc:482] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.181387 14656 raft_consensus.cc:504] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.184389 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19db9cff74843e087fd82db2475c29c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.185017 13058 leader_election.cc:341] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.185513 14656 leader_election.cc:288] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.202381 13244 tablet_bootstrap.cc:492] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.202735 13244 ts_tablet_manager.cc:1367] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.077s user 0.026s sys 0.000s
I20240506 19:57:04.204418 13244 raft_consensus.cc:348] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.204906 14859 tablet_loader.cc:96] loaded metadata for tablet 4592c5bf88b74393ac0ebcf924d5f993 (table default.Table_23e6be51af4a449a8f17e9b5fed56041 [id=9855dc7eecda4d969b9261f88384d088])
I20240506 19:57:04.207079 13244 raft_consensus.cc:374] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.207183 13244 raft_consensus.cc:729] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.207383 13244 consensus_queue.cc:260] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.209467 13244 ts_tablet_manager.cc:1398] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:57:04.210147 13244 tablet_bootstrap.cc:492] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.211977 13244 tablet_bootstrap.cc:654] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.212857 12922 leader_election.cc:302] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.213609 15009 raft_consensus.cc:2781] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.213696 15009 raft_consensus.cc:482] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.213778 15009 raft_consensus.cc:3037] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.216413 15009 raft_consensus.cc:504] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.217799 15009 leader_election.cc:288] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.218331 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee69d8e09c2f4c898ce547ec396ca0e6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.218976 12921 leader_election.cc:341] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.220785 13244 tablet_bootstrap.cc:492] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.221415 13244 ts_tablet_manager.cc:1367] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240506 19:57:04.222817 13244 raft_consensus.cc:348] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.223974 13244 raft_consensus.cc:374] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.224200 13244 raft_consensus.cc:729] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.224650 13244 consensus_queue.cc:260] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.225993 13244 ts_tablet_manager.cc:1398] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240506 19:57:04.227229 13244 tablet_bootstrap.cc:492] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.229887 13244 tablet_bootstrap.cc:654] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.233891 13249 tablet_bootstrap.cc:492] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.234042 13249 ts_tablet_manager.cc:1367] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.237s user 0.004s sys 0.000s
I20240506 19:57:04.234761 13249 raft_consensus.cc:348] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.234967 13249 raft_consensus.cc:374] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.235028 13249 raft_consensus.cc:729] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.235250 13249 consensus_queue.cc:260] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.235935 13249 ts_tablet_manager.cc:1398] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:04.226997 14859 tablet_loader.cc:96] loaded metadata for tablet 45b2a4720f6a46e4b0412a58c1e26bf7 (table default.Table_f659f7ad515e4001891b15f486dc7b1a [id=1ef16dfbd249466fa6b0a7df43515b7b])
I20240506 19:57:04.236603 14859 tablet_loader.cc:96] loaded metadata for tablet 45da091c44824bd6bb8d390a5fb01ef8 (table default.Table_8afdb494805e4ba7838632b12912ec36 [id=22fff2da3af34564b794765f53773f81])
I20240506 19:57:04.237069 14859 tablet_loader.cc:96] loaded metadata for tablet 4630abf54f024da5be5b45efccc0051d (table default.Table_4897151eca6e4c93be0340b5da47fd8c [id=8aff1566821e4751aa6b062b5959f758])
I20240506 19:57:04.238914 15009 consensus_queue.cc:1035] T 4dcc036a20824736b16815bcb47645c4 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.242528 12922 leader_election.cc:302] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.243695 15009 raft_consensus.cc:2781] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.243798 15009 raft_consensus.cc:482] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.243865 15009 raft_consensus.cc:3037] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.246922 15009 raft_consensus.cc:504] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.248098 15009 leader_election.cc:288] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.254016 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.254694 12921 leader_election.cc:341] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.278677 13249 tablet_bootstrap.cc:492] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.280558 14859 tablet_loader.cc:96] loaded metadata for tablet 48769e0dd8864d3b85ee6a3ed1c03c94 (table default.Table_8c4e4e8dbeff40b6b23e5456dbb53729 [id=cb8610d949ac4b1f83b8646f3b7d2caf])
W20240506 19:57:04.281170 13058 consensus_peers.cc:485] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.285398 13244 tablet_bootstrap.cc:492] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.291150 13244 ts_tablet_manager.cc:1367] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.064s user 0.015s sys 0.011s
I20240506 19:57:04.292742 13245 tablet_bootstrap.cc:492] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.293179 13245 ts_tablet_manager.cc:1367] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.290s user 0.033s sys 0.077s
I20240506 19:57:04.293867 13057 leader_election.cc:302] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.294497 15098 raft_consensus.cc:2781] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.294627 15098 raft_consensus.cc:482] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.294819 15098 raft_consensus.cc:3037] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.295147 13245 raft_consensus.cc:348] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.295715 13245 raft_consensus.cc:374] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.295864 13245 raft_consensus.cc:729] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.296417 13245 consensus_queue.cc:260] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.297693 13245 ts_tablet_manager.cc:1398] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240506 19:57:04.304616 12847 raft_consensus.cc:3037] T 2b949c48806d45c3a74efbc8503c0bdb P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.311694 14859 tablet_loader.cc:96] loaded metadata for tablet 491909887be54436bb94162ace8dd78e (table default.Table_e9acd0a6178643ba98580f34636cd25b [id=6ae71958a02d48b384248581d7d91404])
I20240506 19:57:04.312568 13244 raft_consensus.cc:348] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.303416 13249 tablet_bootstrap.cc:654] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.315253 13244 raft_consensus.cc:374] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.317684 14859 tablet_loader.cc:96] loaded metadata for tablet 4973e600038742a6bec0bb2d8f3551fe (table default.Table_c64bdf2c9da94cd3ace74befeae06a29 [id=cd2382012b444dd09cd9ad1a50a20b4c])
I20240506 19:57:04.318588 14859 tablet_loader.cc:96] loaded metadata for tablet 4a369e21e62444208be40d8c552b1626 (table default.Table_0119e18037bb4223ad5b87a035d96386 [id=be8981747042401ebf0f6f583b71d220])
I20240506 19:57:04.319967 14859 tablet_loader.cc:96] loaded metadata for tablet 4be0ceb7d6af4cf4b8c59ddf134ec1ac (table default.Table_cf12763862094c8197c09b72b43f4616 [id=cf9c889d67414705b6bda662c1bc3e8a])
I20240506 19:57:04.321133 14859 tablet_loader.cc:96] loaded metadata for tablet 4bed507b4ac14edb9f4172c0d75a5fd7 (table default.Table_4a6c1aff66ee42a293a7af16d9c5a002 [id=8e8f6ceb668d4e14a5ad41373ab56a4a])
I20240506 19:57:04.316789 13244 raft_consensus.cc:729] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.333165 13249 tablet_bootstrap.cc:492] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.333168 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee69d8e09c2f4c898ce547ec396ca0e6" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:04.333333 13249 ts_tablet_manager.cc:1367] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240506 19:57:04.333451 13120 raft_consensus.cc:3037] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.333994 13249 raft_consensus.cc:348] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.334167 13249 raft_consensus.cc:374] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.334209 13249 raft_consensus.cc:729] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.336521 13120 raft_consensus.cc:2454] T ee69d8e09c2f4c898ce547ec396ca0e6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:04.336759 13249 consensus_queue.cc:260] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.304746 15098 raft_consensus.cc:504] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.331228 13244 consensus_queue.cc:260] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.341858 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:04.342085 13114 raft_consensus.cc:2379] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 059a5c4a3f7e49ef95af4b8341702745 in current term 1: Already voted for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in this term.
W20240506 19:57:04.342494 13058 consensus_peers.cc:485] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.349695 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30a2ce53464d12b98c6a0c2fdeb4fa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:04.350208 13249 ts_tablet_manager.cc:1398] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
W20240506 19:57:04.351034 12781 leader_election.cc:341] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 2a7c61244620487bbbac26dcdbf8a38c
I20240506 19:57:04.351148 12781 leader_election.cc:302] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.351365 14910 raft_consensus.cc:2726] T 2a7c61244620487bbbac26dcdbf8a38c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:04.350342 13058 leader_election.cc:341] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:04.350710 14982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 19:57:04.352077 14982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20240506 19:57:04.349684 15098 leader_election.cc:288] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:04.352553 14982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 19:57:04.352700 12781 leader_election.cc:341] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: da06187c83b94d03ae4ebb67a2873cf0
I20240506 19:57:04.352807 12781 leader_election.cc:302] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.353013 14910 raft_consensus.cc:2726] T da06187c83b94d03ae4ebb67a2873cf0 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.352397 13244 ts_tablet_manager.cc:1398] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.061s user 0.002s sys 0.002s
W20240506 19:57:04.353758 12781 leader_election.cc:341] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: d9443cdbaa3c40299b66d216673c4b1a
I20240506 19:57:04.353855 12781 leader_election.cc:302] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.354233 14910 raft_consensus.cc:2726] T d9443cdbaa3c40299b66d216673c4b1a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.355719 13244 tablet_bootstrap.cc:492] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.362275 13245 tablet_bootstrap.cc:492] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:04.362509 12921 consensus_peers.cc:485] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.363102 12846 raft_consensus.cc:3037] T cade6b1c5a034b23aa33b2349e1dab95 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.364459 13245 tablet_bootstrap.cc:654] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.366946 13249 tablet_bootstrap.cc:492] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.369203 13249 tablet_bootstrap.cc:654] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.376725 13244 tablet_bootstrap.cc:654] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.373890 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8bb09098d401da62ad737ffb697ad" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.375947 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f3def738fb4cb195d08e48d7d2b293" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.377663 12988 raft_consensus.cc:3037] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.377729 12987 raft_consensus.cc:2379] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a9c37db9d4f41cd93820a1c81f48d4d in current term 1: Already voted for candidate 059a5c4a3f7e49ef95af4b8341702745 in this term.
I20240506 19:57:04.378309 14859 tablet_loader.cc:96] loaded metadata for tablet 4c6a2c98fd374b8cbc5a65069bd4cb5b (table default.Table_f282b6bfb0dd40baa082ac8a5ced166e [id=2c018b5df19c4415bf0aa2e45b1f7b97])
I20240506 19:57:04.380327 12988 raft_consensus.cc:2454] T c5f3def738fb4cb195d08e48d7d2b293 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:04.383368 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19db9cff74843e087fd82db2475c29c" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.383662 14859 tablet_loader.cc:96] loaded metadata for tablet 4ca87343f5914e90a1810c62ba87c51a (table default.Table_9ab15c3c5575442181350680a86a5370 [id=7f170fa8d27a423f977115e24d56b038])
I20240506 19:57:04.385475 14859 tablet_loader.cc:96] loaded metadata for tablet 4d09e7b64b6d45618c6595c1013922df (table default.Table_8d7fd547dfb74b66a34d4207ef4bf800 [id=01881436327441a384925f571f99bf14])
I20240506 19:57:04.389236 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30a2ce53464d12b98c6a0c2fdeb4fa" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.389420 12987 raft_consensus.cc:3037] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.392911 12987 raft_consensus.cc:2454] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
W20240506 19:57:04.396906 13058 consensus_peers.cc:485] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.398429 14859 tablet_loader.cc:96] loaded metadata for tablet 4d12edf17784486b9c598c2f29675060 (table default.Table_f9f5211355fa4a10b23a184e7c55ae7c [id=e311f1c887124c15a887b9def426858e])
I20240506 19:57:04.398790 13244 tablet_bootstrap.cc:492] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.400310 13244 ts_tablet_manager.cc:1367] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.001s
I20240506 19:57:04.401917 14859 tablet_loader.cc:96] loaded metadata for tablet 4e2c9f4baf55412d9bc00818a00c652d (table default.Table_ecb75e9b5c5d4c4ebd6f3684a6a350f9 [id=034d7824127b4f94aa24bdeb76bfbd7c])
I20240506 19:57:04.402824 13244 raft_consensus.cc:348] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.398175 13249 tablet_bootstrap.cc:492] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.403885 13249 ts_tablet_manager.cc:1367] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20240506 19:57:04.404541 13249 raft_consensus.cc:348] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.404727 13249 raft_consensus.cc:374] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.404784 13249 raft_consensus.cc:729] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.404636 15098 consensus_queue.cc:1035] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.405025 13249 consensus_queue.cc:260] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.406431 14656 consensus_queue.cc:1035] T 9944e5bfe0a741d69b8f9b6a9dad84dd P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.418802 15011 raft_consensus.cc:482] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.418951 15011 raft_consensus.cc:504] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.419535 15011 leader_election.cc:288] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.421470 13249 ts_tablet_manager.cc:1398] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240506 19:57:04.421856 13249 tablet_bootstrap.cc:492] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.432801 13245 tablet_bootstrap.cc:492] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.432955 13245 ts_tablet_manager.cc:1367] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.002s
W20240506 19:57:04.433079 13058 consensus_peers.cc:485] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.433817 13245 raft_consensus.cc:348] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.434038 13245 raft_consensus.cc:374] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.434103 13245 raft_consensus.cc:729] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.434304 13245 consensus_queue.cc:260] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.435034 13245 ts_tablet_manager.cc:1398] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:04.435304 13245 tablet_bootstrap.cc:492] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:04.436771 13057 leader_election.cc:302] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.437237 15098 raft_consensus.cc:2726] T 6fb8bb09098d401da62ad737ffb697ad P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.420269 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4e628b9364cccb525b8e1b8070286" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.437942 12921 leader_election.cc:341] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.438378 13249 tablet_bootstrap.cc:654] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.438095 13244 raft_consensus.cc:374] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.439368 13057 leader_election.cc:302] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.439822 15098 raft_consensus.cc:2781] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.440205 15098 raft_consensus.cc:686] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.440495 15098 consensus_queue.cc:237] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.439054 13244 raft_consensus.cc:729] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
W20240506 19:57:04.444033 13057 leader_election.cc:341] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.444139 13057 leader_election.cc:302] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.444535 14656 raft_consensus.cc:2726] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.446290 14859 tablet_loader.cc:96] loaded metadata for tablet 4e7cc9769ad84cf6af3cfc0a45d63aa9 (table default.Table_d529a0b0f9774e0385d3b2319db222dc [id=16aa3036ed744751afd08515bb792ce2])
I20240506 19:57:04.445852 13244 consensus_queue.cc:260] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.448001 14859 tablet_loader.cc:96] loaded metadata for tablet 4ea8f5301f524e4e8910ce61837a1dd4 (table default.Table_300911c21c1140aca1cd2170583017fb [id=6aecb90966cd41f9aa1066cd32246615])
I20240506 19:57:04.450331 13057 leader_election.cc:302] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.450500 14656 consensus_queue.cc:1035] T 09cb71b5962f4561bf0fdbe12bf45296 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.450770 14915 raft_consensus.cc:2781] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.450914 14915 raft_consensus.cc:686] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.451186 14915 consensus_queue.cc:237] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.452096 13245 tablet_bootstrap.cc:654] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.453572 13249 tablet_bootstrap.cc:492] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.453769 13249 ts_tablet_manager.cc:1367] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240506 19:57:04.454279 13244 ts_tablet_manager.cc:1398] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20240506 19:57:04.454921 14859 tablet_loader.cc:96] loaded metadata for tablet 4ed98067bef848349d9dd93805674bbd (table default.Table_fb84f06ac8d34f659ca376315100395f [id=8b3fcd8705044aba983c11196817167f])
I20240506 19:57:04.455212 15011 raft_consensus.cc:482] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.455358 15011 raft_consensus.cc:504] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.456051 15011 leader_election.cc:288] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.456264 13249 raft_consensus.cc:348] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.456509 13249 raft_consensus.cc:374] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.456570 13249 raft_consensus.cc:729] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.456821 13249 consensus_queue.cc:260] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.458580 13249 ts_tablet_manager.cc:1398] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:57:04.462525 13249 tablet_bootstrap.cc:492] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.464707 13249 tablet_bootstrap.cc:654] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.456732 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3291123a1e42af80dbfb578c855618" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.469429 12921 leader_election.cc:341] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:04.450378 13058 consensus_peers.cc:485] T 597a48f6950844299f5f292e6a101cae P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:04.472559 15011 consensus_queue.cc:1035] T a03f574aec984b83ab596e92a6f323fc P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.474771 13244 tablet_bootstrap.cc:492] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.475299 14859 tablet_loader.cc:96] loaded metadata for tablet 4f0c085734de48a0a51f43c4e5ddf77d (table default.Table_8dc9c0fe4f3d410f94f4c5ac9071fc20 [id=e9d56f7b210b48caac7ad8b211e7de13])
I20240506 19:57:04.481173 14859 tablet_loader.cc:96] loaded metadata for tablet 5024b8379db74d278f91894fad290691 (table default.Table_6fb4b0e1d5c94d88b7f9792005908344 [id=0206d0193f404359bf21bc2d8662c3c0])
I20240506 19:57:04.481465 12922 leader_election.cc:302] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.486198 13249 tablet_bootstrap.cc:492] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.486353 13249 ts_tablet_manager.cc:1367] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.001s
I20240506 19:57:04.486511 15011 raft_consensus.cc:2781] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.486651 15011 raft_consensus.cc:686] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:04.486929 15011 consensus_queue.cc:237] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.487141 13249 raft_consensus.cc:348] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.487322 13249 raft_consensus.cc:374] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.487377 13249 raft_consensus.cc:729] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.487536 13249 consensus_queue.cc:260] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.487596 12922 leader_election.cc:302] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.488260 15248 raft_consensus.cc:2726] T 6fb8bb09098d401da62ad737ffb697ad P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.488822 13249 ts_tablet_manager.cc:1398] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 19:57:04.489092 13249 tablet_bootstrap.cc:492] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.489840 14915 raft_consensus.cc:482] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.489984 14915 raft_consensus.cc:504] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.491107 13249 tablet_bootstrap.cc:654] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.491425 14915 leader_election.cc:288] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.491979 14797 consensus_queue.cc:1035] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.492602 14797 consensus_queue.cc:1035] T 7a396afff3b243ef84cb3dc3836fcdfd P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.495811 15098 raft_consensus.cc:482] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.496023 15098 raft_consensus.cc:504] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.496898 15098 leader_election.cc:288] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.496376 14859 tablet_loader.cc:96] loaded metadata for tablet 51ab070924f3472f8f8301920f521503 (table default.Table_34e944977e794f829c328e4fc138baec [id=da72f500d27941dd826d6896b02cbeeb])
I20240506 19:57:04.498215 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4e628b9364cccb525b8e1b8070286" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.498467 13120 raft_consensus.cc:2454] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:04.491317 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c652feb405ef4f1cab8ee602e9d3eb18" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:04.505131 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8321457717fd4ad28ae1292d9e9a8669" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.506273 13058 leader_election.cc:341] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:04.506528 13058 leader_election.cc:341] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.509969 12846 raft_consensus.cc:3037] T 76e6f7555dfa4122ac00bae7b9e733fe P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.510860 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3291123a1e42af80dbfb578c855618" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.511416 13120 raft_consensus.cc:2454] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:04.525707 14859 tablet_loader.cc:96] loaded metadata for tablet 530de4f3f9584ea3b695e3abc08ad560 (table default.Table_b2e9c53dcdb14928b637180cdca5f786 [id=3b6c71a27559456aa2d11f0736cf5dbf])
I20240506 19:57:04.527827 13244 tablet_bootstrap.cc:654] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.534963 14859 tablet_loader.cc:96] loaded metadata for tablet 5553e49640594e258d3506b9250a6c9f (table default.Table_e63955b8a5324e6688d4a2126f9a2315 [id=5eae429c4d3c4a199359ea1c6532b6d6])
I20240506 19:57:04.537106 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c652feb405ef4f1cab8ee602e9d3eb18" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.538623 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8321457717fd4ad28ae1292d9e9a8669" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.538827 12983 raft_consensus.cc:2454] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:04.543150 13245 tablet_bootstrap.cc:492] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.543421 13245 ts_tablet_manager.cc:1367] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.003s
I20240506 19:57:04.542739 14859 tablet_loader.cc:96] loaded metadata for tablet 565d5f6225a74c4da1aba61d30e2a8de (table default.Table_b9e1af536a204e51a898c9821976d7ac [id=4a489bb59b9545e991edc010e270af80])
I20240506 19:57:04.544410 13245 raft_consensus.cc:348] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.544610 13245 raft_consensus.cc:374] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.544677 13245 raft_consensus.cc:729] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.545038 13245 consensus_queue.cc:260] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.545737 13245 ts_tablet_manager.cc:1398] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:04.546000 13245 tablet_bootstrap.cc:492] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:04.550657 13245 tablet_bootstrap.cc:654] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:04.550849 13057 leader_election.cc:341] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.550981 13057 leader_election.cc:302] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: 059a5c4a3f7e49ef95af4b8341702745, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.551581 15098 raft_consensus.cc:2726] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.549764 14859 tablet_loader.cc:96] loaded metadata for tablet 567cb184adf14b78aaf12d0bbdcf7826 (table default.Table_0921ada34d224bd7b8bc59edc8627549 [id=14b3734b28354d60b48fa59014220a25])
I20240506 19:57:04.553253 13057 leader_election.cc:302] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.553877 15098 raft_consensus.cc:2781] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.554011 15098 raft_consensus.cc:482] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.554149 15098 raft_consensus.cc:3037] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.557797 15098 raft_consensus.cc:504] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.562130 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8321457717fd4ad28ae1292d9e9a8669" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:04.562274 15098 leader_election.cc:288] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:04.563781 13058 leader_election.cc:341] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.564762 14859 tablet_loader.cc:96] loaded metadata for tablet 56de924d3f104a7da4f4c37924875555 (table default.Table_6449ebda2e1b4b0088a1c4bd083cf123 [id=2126bfc39a3b4b34b6edf5460144fb46])
I20240506 19:57:04.565398 12922 leader_election.cc:302] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.570283 15248 raft_consensus.cc:2781] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.570437 15248 raft_consensus.cc:482] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.570596 15248 raft_consensus.cc:3037] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.573443 15248 raft_consensus.cc:504] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.574170 15248 leader_election.cc:288] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.574570 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4e628b9364cccb525b8e1b8070286" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.575034 12921 leader_election.cc:341] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.575798 12922 leader_election.cc:302] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.576583 14859 tablet_loader.cc:96] loaded metadata for tablet 5751124adb5b40f59e8d79a4605ae8c4 (table default.Table_ece1e5630f42487eb58e4d048c7bbc54 [id=2c295d2600344234889c7ad8019bd164])
I20240506 19:57:04.576934 15248 raft_consensus.cc:2781] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.577091 15248 raft_consensus.cc:482] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.577225 15248 raft_consensus.cc:3037] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.579792 15248 raft_consensus.cc:504] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.580715 15248 leader_election.cc:288] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.581116 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3291123a1e42af80dbfb578c855618" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:04.588790 14859 tablet_loader.cc:96] loaded metadata for tablet 57a58ed1c4bd4f80834f9c6f52355334 (table default.Table_78979d94cf774d05a800befc0a5b9745 [id=5c6305fd4cde4b1a87ae85c9267d759e])
W20240506 19:57:04.581681 12921 leader_election.cc:341] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.591611 13244 tablet_bootstrap.cc:492] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.592458 13244 ts_tablet_manager.cc:1367] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.000s
I20240506 19:57:04.593715 13244 raft_consensus.cc:348] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.594728 13249 tablet_bootstrap.cc:492] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.594949 13249 ts_tablet_manager.cc:1367] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.002s
I20240506 19:57:04.595736 13249 raft_consensus.cc:348] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.596015 13249 raft_consensus.cc:374] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.596128 13249 raft_consensus.cc:729] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.596433 13249 consensus_queue.cc:260] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.597108 13249 ts_tablet_manager.cc:1398] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:04.597455 13249 tablet_bootstrap.cc:492] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.602183 13244 raft_consensus.cc:374] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.603076 13249 tablet_bootstrap.cc:654] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.603567 13244 raft_consensus.cc:729] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.604365 13244 consensus_queue.cc:260] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.606818 14859 tablet_loader.cc:96] loaded metadata for tablet 582725a8a48546a68937f7d11a1873a4 (table default.Table_16507180ad3142ee968513548406bd0b [id=02a9361327124ac9a5359df5020f59e4])
I20240506 19:57:04.607401 15098 consensus_queue.cc:1035] T 4216f98cb6ac4e249ca2653518ff9e87 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.610177 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4e628b9364cccb525b8e1b8070286" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:04.610356 13116 raft_consensus.cc:3037] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:04.610994 14982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 19:57:04.611110 14982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 19:57:04.611171 14982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 19:57:04.611236 14982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 19:57:04.612313 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3291123a1e42af80dbfb578c855618" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:04.612501 13120 raft_consensus.cc:3037] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.613058 13116 raft_consensus.cc:2454] T 4be4e628b9364cccb525b8e1b8070286 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:04.615521 13120 raft_consensus.cc:2454] T 2a3291123a1e42af80dbfb578c855618 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:04.619544 14859 tablet_loader.cc:96] loaded metadata for tablet 597a48f6950844299f5f292e6a101cae (table default.Table_4cc2d777e8ad43d49ca9092206e8dc72 [id=74dfbbbb284d48a8bb4fb2626eb67a3b])
I20240506 19:57:04.620430 14859 tablet_loader.cc:96] loaded metadata for tablet 597c61f1d1104256b90f6601cf3e6667 (table default.Table_715977f202b24ca7a07c82ef612e23bb [id=ef86771f67d14d9090bcc7722a834478])
I20240506 19:57:04.623656 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8321457717fd4ad28ae1292d9e9a8669" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.624392 12988 raft_consensus.cc:3037] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.628024 12988 raft_consensus.cc:2454] T 8321457717fd4ad28ae1292d9e9a8669 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:04.634405 13244 ts_tablet_manager.cc:1398] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240506 19:57:04.634644 14700 raft_consensus.cc:482] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.634784 14700 raft_consensus.cc:504] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.635386 14700 leader_election.cc:288] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.636126 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eac98ac8f040c58f61106bb0e989de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.636886 12921 leader_election.cc:341] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.637838 13244 tablet_bootstrap.cc:492] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.639779 12846 raft_consensus.cc:3037] T 0fb9c05d703f417780e77359e19e542e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.648838 14700 raft_consensus.cc:482] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.649040 14910 raft_consensus.cc:482] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.648991 14700 raft_consensus.cc:504] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.649176 14910 raft_consensus.cc:504] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.649577 14700 leader_election.cc:288] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.649832 14910 leader_election.cc:288] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:04.650592 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.650833 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8f5301f524e4e8910ce61837a1dd4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.651494 12921 leader_election.cc:341] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.651696 13249 tablet_bootstrap.cc:492] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.651839 13249 ts_tablet_manager.cc:1367] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20240506 19:57:04.660784 13245 tablet_bootstrap.cc:492] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.660951 13245 ts_tablet_manager.cc:1367] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.001s
I20240506 19:57:04.661803 13245 raft_consensus.cc:348] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.662019 13245 raft_consensus.cc:374] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.662083 13245 raft_consensus.cc:729] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.662328 13245 consensus_queue.cc:260] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.662127 13249 raft_consensus.cc:348] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.664719 13057 leader_election.cc:302] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.665725 14656 raft_consensus.cc:2781] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.665885 14656 raft_consensus.cc:686] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.666162 14656 consensus_queue.cc:237] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.667264 15103 consensus_queue.cc:1035] T 35453f75a3494f8baf13e0d5cacc9003 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:04.668699 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eac98ac8f040c58f61106bb0e989de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.668902 13120 raft_consensus.cc:2454] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:04.668947 13249 raft_consensus.cc:374] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.669041 13249 raft_consensus.cc:729] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.669555 14859 tablet_loader.cc:96] loaded metadata for tablet 599f6a7b76d14c9589e5cabb04bb5f4c (table default.Table_c35eaf4f2426412381b537c1bcb048f7 [id=8439d7bbb943423f9c433fd0c82f1773])
I20240506 19:57:04.669591 13249 consensus_queue.cc:260] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.670145 13249 ts_tablet_manager.cc:1398] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240506 19:57:04.670404 13249 tablet_bootstrap.cc:492] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.674086 14859 tablet_loader.cc:96] loaded metadata for tablet 5a3bd92908f341d99f8756a654799019 (table default.Table_51d50ac0d42d4388a250ec5fd84128a8 [id=0f8c1a749dd94feb907921bc2524dbc2])
I20240506 19:57:04.678179 13244 tablet_bootstrap.cc:654] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.680392 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8d4891c484627ac729971ff7aad3e" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.680471 12922 leader_election.cc:302] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.684450 15248 raft_consensus.cc:2781] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.684587 15248 raft_consensus.cc:686] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:04.684623 12922 leader_election.cc:302] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.684834 15248 consensus_queue.cc:237] T 4be4e628b9364cccb525b8e1b8070286 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.685266 14859 tablet_loader.cc:96] loaded metadata for tablet 5a7aa1324e9749948c311f5c291ff06d (table default.Table_13e6e7d68e57433ebed02111652c43a9 [id=fe195172a40d45a5aef9388abdc14672])
I20240506 19:57:04.685402 13249 tablet_bootstrap.cc:654] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.686004 14859 tablet_loader.cc:96] loaded metadata for tablet 5b375490272a4dec9a743f6458f6d6de (table default.Table_ab5b8e26274d4c17aafc75dabab5539a [id=c749aef4b9e0450fada39042d70c1917])
I20240506 19:57:04.686457 14859 tablet_loader.cc:96] loaded metadata for tablet 5b472ba7729c4f6184e07eff618407f4 (table default.Table_7cbfc903cd8e400e87d5f107413810f6 [id=9808065d63b34ce5963ba728f22b3772])
I20240506 19:57:04.687197 14859 tablet_loader.cc:96] loaded metadata for tablet 5ca4db0191084409a90e04c65afd8e55 (table default.Table_3380422dfa66489eb9f22c0b76154c8c [id=31bc34780203473fbb7aa8f32487c8fa])
I20240506 19:57:04.687639 14859 tablet_loader.cc:96] loaded metadata for tablet 5cee27bc6e334dc3ae25f7ce5c40da00 (table default.Table_a0d74db25f05423f8b5ae3b82c7a54b1 [id=21294b97805743eb8444823b07a0ed7b])
I20240506 19:57:04.688009 14859 tablet_loader.cc:96] loaded metadata for tablet 5d06b7c1428340638f585f195fc4d583 (table default.Table_02ef9cd7cbba4ec1bc2971e5c8f43cda [id=ef25759917464cb5a3c0feb3f384c417])
I20240506 19:57:04.689047 14859 tablet_loader.cc:96] loaded metadata for tablet 5e09d48fbdb5490bbbf58d0daee40b14 (table default.Table_8c2dd5f0bc85416b9c98a687c245f5b5 [id=62473581d9f442d993eb190d59c90efe])
I20240506 19:57:04.690407 15009 raft_consensus.cc:2781] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.690555 15009 raft_consensus.cc:686] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:04.690758 15009 consensus_queue.cc:237] T 2a3291123a1e42af80dbfb578c855618 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.692386 12922 leader_election.cc:302] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.693169 13244 tablet_bootstrap.cc:492] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.693372 13244 ts_tablet_manager.cc:1367] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.000s
I20240506 19:57:04.693133 15011 raft_consensus.cc:2781] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.693735 15011 raft_consensus.cc:482] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.694160 12781 leader_election.cc:302] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.694464 14910 raft_consensus.cc:2726] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.694335 15011 raft_consensus.cc:3037] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.694180 13244 raft_consensus.cc:348] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.694895 13244 raft_consensus.cc:374] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.695011 13244 raft_consensus.cc:729] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.695286 13244 consensus_queue.cc:260] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.696839 13244 ts_tablet_manager.cc:1398] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240506 19:57:04.697813 13244 tablet_bootstrap.cc:492] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.699491 13245 ts_tablet_manager.cc:1398] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20240506 19:57:04.701324 13244 tablet_bootstrap.cc:654] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.705407 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8f5301f524e4e8910ce61837a1dd4" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.705705 13249 tablet_bootstrap.cc:492] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.705834 13249 ts_tablet_manager.cc:1367] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20240506 19:57:04.706759 13249 raft_consensus.cc:348] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.706950 13249 raft_consensus.cc:374] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.707006 13249 raft_consensus.cc:729] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.707226 13249 consensus_queue.cc:260] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.707875 13249 ts_tablet_manager.cc:1398] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:04.708146 13249 tablet_bootstrap.cc:492] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.709786 13249 tablet_bootstrap.cc:654] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.713869 13244 tablet_bootstrap.cc:492] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.714092 13244 ts_tablet_manager.cc:1367] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240506 19:57:04.714891 13244 raft_consensus.cc:348] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.715240 13244 raft_consensus.cc:374] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.715430 13244 raft_consensus.cc:729] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.715713 13244 consensus_queue.cc:260] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.715934 13245 tablet_bootstrap.cc:492] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:04.716733 13244 ts_tablet_manager.cc:1398] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:04.717264 13244 tablet_bootstrap.cc:492] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.720341 13244 tablet_bootstrap.cc:654] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:04.723253 12922 leader_election.cc:341] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.721846 14915 consensus_queue.cc:1035] T 58c88f80f5b34672b7c3a4b604469bf1 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240506 19:57:04.728813 13244 tablet_bootstrap.cc:492] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.729455 13244 ts_tablet_manager.cc:1367] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20240506 19:57:04.729588 12922 leader_election.cc:302] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.729787 13245 tablet_bootstrap.cc:654] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.730489 15010 raft_consensus.cc:2726] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.730468 13244 raft_consensus.cc:348] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.727653 15011 raft_consensus.cc:504] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.739964 14859 tablet_loader.cc:96] loaded metadata for tablet 5ec23d785839462b91c036705cac2a72 (table default.Table_b879654fd2304197a107fc833fd81e21 [id=c9ddd47c049f435eafb85d7bd033cc2e])
I20240506 19:57:04.740785 13244 raft_consensus.cc:374] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.741140 13244 raft_consensus.cc:729] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.741603 13244 consensus_queue.cc:260] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.743181 14859 tablet_loader.cc:96] loaded metadata for tablet 6089f4f68bfe415e8d97dfd6849b7714 (table default.Table_15b63528dc5649619d25b71595773037 [id=30577c164a974b99a479276640cca87e])
I20240506 19:57:04.746507 14859 tablet_loader.cc:96] loaded metadata for tablet 6097fe368a894f648b097912e16e361f (table default.Table_179cf5eeca0d46ee9128f0c4c1bcc5f8 [id=f5816273b8914052a9b58d5495bf45a1])
I20240506 19:57:04.753858 14859 tablet_loader.cc:96] loaded metadata for tablet 60f07ed7653f4910b3a47890842216bc (table default.Table_10658c410e1648df879bdf21d2293b2e [id=b4856100088a41d083991b82b8185f1c])
I20240506 19:57:04.756196 14656 raft_consensus.cc:482] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.756388 15011 leader_election.cc:288] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.756448 14656 raft_consensus.cc:504] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.757829 14656 leader_election.cc:288] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.756191 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eac98ac8f040c58f61106bb0e989de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:04.757915 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb1559fe135498f9fefbbf04daa6909" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.761762 12921 leader_election.cc:341] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:04.762475 13058 leader_election.cc:341] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.762676 13244 ts_tablet_manager.cc:1398] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20240506 19:57:04.763998 13244 tablet_bootstrap.cc:492] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.764636 14859 tablet_loader.cc:96] loaded metadata for tablet 61d8da431d09460ea4035343a15826ec (table default.Table_d2889007e246431bb796c67ca72db0aa [id=63af27bdf8c84d478759499c8437682c])
I20240506 19:57:04.765807 13245 tablet_bootstrap.cc:492] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.765973 13245 ts_tablet_manager.cc:1367] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.006s
I20240506 19:57:04.766077 14910 raft_consensus.cc:482] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.766220 14910 raft_consensus.cc:504] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.766844 14910 leader_election.cc:288] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:04.767365 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21059ce22bff4ccc8ae7d0de0c2fbd7a" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.766760 13245 raft_consensus.cc:348] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.767535 13245 raft_consensus.cc:374] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.767653 13245 raft_consensus.cc:729] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.768005 13245 consensus_queue.cc:260] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.768424 13244 tablet_bootstrap.cc:654] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.769320 14859 tablet_loader.cc:96] loaded metadata for tablet 62f7ed8e26b949028372656f2bbcc8f6 (table default.Table_0b1a8466d287416ab4ec1f38a4687e69 [id=a7fd74ddf9f24e12bdb816bdd9c65fd2])
I20240506 19:57:04.770097 14859 tablet_loader.cc:96] loaded metadata for tablet 6352d2c630204df3a24e299e167f92d4 (table default.Table_1a5a91601a7c4cc3ae0b43e0f3761e17 [id=c2803598df8241beb0b6ce684515661c])
I20240506 19:57:04.771806 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eac98ac8f040c58f61106bb0e989de" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:04.771977 13114 raft_consensus.cc:3037] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.774663 13114 raft_consensus.cc:2454] T b6eac98ac8f040c58f61106bb0e989de P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:04.774974 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb1559fe135498f9fefbbf04daa6909" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.775156 12988 raft_consensus.cc:2454] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:04.776104 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21059ce22bff4ccc8ae7d0de0c2fbd7a" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.777827 14859 tablet_loader.cc:96] loaded metadata for tablet 63decce057244d72b4a812368594fcfb (table default.Table_de7835c60b14419892d8827447c27f2e [id=561add2e304a497da06ce1abc6054980])
I20240506 19:57:04.778961 13057 leader_election.cc:302] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.779743 14915 raft_consensus.cc:2781] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:04.779825 14915 raft_consensus.cc:482] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.779884 14915 raft_consensus.cc:3037] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.779958 12781 leader_election.cc:302] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.780221 14910 raft_consensus.cc:2726] T 21059ce22bff4ccc8ae7d0de0c2fbd7a P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.781234 13245 ts_tablet_manager.cc:1398] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240506 19:57:04.781160 12922 leader_election.cc:302] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.781860 15009 raft_consensus.cc:2781] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.781973 15009 raft_consensus.cc:686] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [term 1 LEADER]: Becoming Leader. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Running, Role: LEADER
I20240506 19:57:04.782188 15009 consensus_queue.cc:237] T b6eac98ac8f040c58f61106bb0e989de P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.782428 14915 raft_consensus.cc:504] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.782984 14915 leader_election.cc:288] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.784839 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb1559fe135498f9fefbbf04daa6909" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.785583 13058 leader_election.cc:341] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.788865 15268 raft_consensus.cc:482] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.788996 15268 raft_consensus.cc:504] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.789651 15268 leader_election.cc:288] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:04.790280 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.792088 13249 tablet_bootstrap.cc:492] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.792270 13249 ts_tablet_manager.cc:1367] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.004s
I20240506 19:57:04.793061 13249 raft_consensus.cc:348] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.793346 13249 raft_consensus.cc:374] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.793443 13249 raft_consensus.cc:729] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.793697 13249 consensus_queue.cc:260] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.794268 13249 ts_tablet_manager.cc:1398] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 19:57:04.798285 13249 tablet_bootstrap.cc:492] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.799952 13249 tablet_bootstrap.cc:654] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.800645 14859 tablet_loader.cc:96] loaded metadata for tablet 63e29e673a784056a79df0b73677f38a (table default.Table_f76cd2f478bd458697b5873a16fbb55b [id=ef23a1148e3648f49b43c73a89b6e598])
I20240506 19:57:04.803041 14982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.48.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.48.254:33495,127.12.48.253:36819,127.12.48.252:36867
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.48.254:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.48.254
--webserver_port=40249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQMIkBc/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715025388288849-12483-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 19:43:22 UTC on efd1337a2aa3
build id 167
ASAN enabled
I20240506 19:57:04.806522 13249 tablet_bootstrap.cc:492] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.806763 13249 ts_tablet_manager.cc:1367] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240506 19:57:04.807914 14982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 19:57:04.808120 14859 tablet_loader.cc:96] loaded metadata for tablet 646fa0d9bd7140be95eef34839fe8acb (table default.Table_91cb9ddcef194b72bf0c1dcbfdab9ef5 [id=7a0f775e3b564c95bd5b8a1139facf87])
I20240506 19:57:04.808961 14859 tablet_loader.cc:96] loaded metadata for tablet 64bc79ad98bf4dce9d3c2dd0517d44a2 (table default.Table_a2366efe03f84bdcabf1d6d62fc25ec7 [id=d5d0e57991d34f01822c91354809b6aa])
I20240506 19:57:04.809671 14859 tablet_loader.cc:96] loaded metadata for tablet 64fabc03ad114a80a236a629db6fd311 (table default.Table_7b12e2d9a5d0496586f34ba591615326 [id=fab32d0c08604ab7976f3c3ef48e4859])
I20240506 19:57:04.810324 14859 tablet_loader.cc:96] loaded metadata for tablet 65315d8ac5ff408bae8593681335e5b1 (table default.Table_712005444bfb41819b6c7f2e3a33f723 [id=7e6330d3765f4fb4bc736225a45862f3])
I20240506 19:57:04.810477 14982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 19:57:04.810822 13245 tablet_bootstrap.cc:492] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:04.812495 13245 tablet_bootstrap.cc:654] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.812569 13249 raft_consensus.cc:348] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.812767 13249 raft_consensus.cc:374] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.812827 13249 raft_consensus.cc:729] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.813024 13249 consensus_queue.cc:260] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.813329 15011 raft_consensus.cc:482] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.813441 15011 raft_consensus.cc:504] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.814044 15011 leader_election.cc:288] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.815398 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee1a4a3f060407ca9c33467474e4489" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.816002 12921 leader_election.cc:341] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.817065 13249 ts_tablet_manager.cc:1398] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240506 19:57:04.818277 13249 tablet_bootstrap.cc:492] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.820276 13249 tablet_bootstrap.cc:654] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.820637 12847 raft_consensus.cc:3037] T 5b375490272a4dec9a743f6458f6d6de P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.826246 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb1559fe135498f9fefbbf04daa6909" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.826448 12986 raft_consensus.cc:3037] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.829185 12986 raft_consensus.cc:2454] T 1bb1559fe135498f9fefbbf04daa6909 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:04.835083 13244 tablet_bootstrap.cc:492] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.835314 13244 ts_tablet_manager.cc:1367] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.011s
I20240506 19:57:04.835561 13249 tablet_bootstrap.cc:492] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.835713 13249 ts_tablet_manager.cc:1367] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240506 19:57:04.836304 13244 raft_consensus.cc:348] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.836443 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fed2f12eda54be0879b1e9c33ca3e31" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.836457 13249 raft_consensus.cc:348] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.836639 13249 raft_consensus.cc:374] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.836694 13249 raft_consensus.cc:729] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.836675 13244 raft_consensus.cc:374] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.836859 13244 raft_consensus.cc:729] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.836916 13249 consensus_queue.cc:260] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.837215 13244 consensus_queue.cc:260] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.838152 13244 ts_tablet_manager.cc:1398] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 19:57:04.838205 13249 ts_tablet_manager.cc:1398] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:04.838515 13244 tablet_bootstrap.cc:492] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.839645 13249 tablet_bootstrap.cc:492] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.840876 14859 tablet_loader.cc:96] loaded metadata for tablet 65b77b8c5c1a4737b555898ae642a578 (table default.Table_eebcdd01355340739cb45d1acca7daa6 [id=ff67c807ba92484dbda8d03d7df13c6e])
I20240506 19:57:04.841475 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee1a4a3f060407ca9c33467474e4489" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.842541 13244 tablet_bootstrap.cc:654] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.844336 13057 leader_election.cc:302] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.845028 15104 raft_consensus.cc:2781] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.845153 15104 raft_consensus.cc:686] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.845399 15104 consensus_queue.cc:237] T 1bb1559fe135498f9fefbbf04daa6909 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.845875 12781 leader_election.cc:302] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.846267 15268 raft_consensus.cc:2726] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.852098 14859 tablet_loader.cc:96] loaded metadata for tablet 66da88df0ab54047b31f871015acfde9 (table default.Table_b1b710ee5f9941a08a8db2bd7122ef25 [id=a34f8af686ae415c95fa33cbf9fef7bc])
W20240506 19:57:04.852052 12922 leader_election.cc:341] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.852782 12922 leader_election.cc:302] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745; no voters: 8a9c37db9d4f41cd93820a1c81f48d4d, ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.854249 13244 tablet_bootstrap.cc:492] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.854348 15011 raft_consensus.cc:2726] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.854465 13244 ts_tablet_manager.cc:1367] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240506 19:57:04.855199 13244 raft_consensus.cc:348] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.855476 13244 raft_consensus.cc:374] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.855577 13244 raft_consensus.cc:729] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.855866 13244 consensus_queue.cc:260] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.861948 13244 ts_tablet_manager.cc:1398] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240506 19:57:04.863319 13244 tablet_bootstrap.cc:492] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.863726 14859 tablet_loader.cc:96] loaded metadata for tablet 68027db821bf4ad3beffc7d160c98988 (table default.Table_54845e03ef4e4cc7b472aeee0c2b4c42 [id=2e805268d6aa44a681a952f654d3055a])
I20240506 19:57:04.865865 13245 tablet_bootstrap.cc:492] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.866014 13245 ts_tablet_manager.cc:1367] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.004s
I20240506 19:57:04.869092 14859 tablet_loader.cc:96] loaded metadata for tablet 6868b3aee874401aa6d7e893446988f7 (table default.Table_728d3e1c6e2f499fbbd7bc0994b341a3 [id=63fb5dfe8fe64ae199a4861c0a175152])
I20240506 19:57:04.870060 14859 tablet_loader.cc:96] loaded metadata for tablet 689901ddfa9f492f98b67daf000a28ae (table default.Table_82ad0ca5d8e2425b98dc9d534a3436cc [id=e4c63adb11504619b37bdc6c1e025c36])
I20240506 19:57:04.871098 14859 tablet_loader.cc:96] loaded metadata for tablet 6a077a0ae1f4411c9fb2409311a9d9ba (table default.Table_6d330fca725b4de09fa2e8a21d67d676 [id=436998b2930242f8ad542a3a8ae6a91c])
I20240506 19:57:04.872952 13245 raft_consensus.cc:348] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.873152 13245 raft_consensus.cc:374] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.873220 13245 raft_consensus.cc:729] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.874608 13245 consensus_queue.cc:260] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.875126 13245 ts_tablet_manager.cc:1398] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240506 19:57:04.876686 13245 tablet_bootstrap.cc:492] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
W20240506 19:57:04.877030 15350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:57:04.880335 13249 tablet_bootstrap.cc:654] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.872180 14859 tablet_loader.cc:96] loaded metadata for tablet 6ace0ec43ba44060bcec2a50e1f48098 (table default.Table_61becbd043ee470ca2ee3f4969d59971 [id=fcfc5bd1da1b4625a2b0ebee1207abdd])
I20240506 19:57:04.882048 13244 tablet_bootstrap.cc:654] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.884157 14859 tablet_loader.cc:96] loaded metadata for tablet 6c89b80216d640ddbf65956fbd714fc3 (table default.Table_131e6a1d2bc240439badeffed52cd489 [id=2a02c78fdbd1463987715da7a840f0ce])
I20240506 19:57:04.885272 14859 tablet_loader.cc:96] loaded metadata for tablet 6ceb1282a53d49ccb11b1c7239e2b364 (table default.Table_bdd8b0bb2f654920a49117008a662af1 [id=f0c703ff6122447b929a5e49ac7f3f42])
I20240506 19:57:04.886336 14859 tablet_loader.cc:96] loaded metadata for tablet 6d815013b9f54acb8a978137961921a1 (table default.Table_2a2d910a263840e7934d2972c9cd8a87 [id=0f8f012e7cf24efe9c4a08abf3fab067])
I20240506 19:57:04.887522 14859 tablet_loader.cc:96] loaded metadata for tablet 6e080ee371c7405fb78faf0d77c56a9d (table default.Table_65858cfb30f343b5ae7629f09d0883b0 [id=bbe4d5d027284555846908c5b7b5eb88])
I20240506 19:57:04.884752 13245 tablet_bootstrap.cc:654] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.898535 13245 tablet_bootstrap.cc:492] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:04.898691 13245 ts_tablet_manager.cc:1367] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240506 19:57:04.899427 13245 raft_consensus.cc:348] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.899619 13245 raft_consensus.cc:374] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.899675 13245 raft_consensus.cc:729] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.899873 13245 consensus_queue.cc:260] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.900377 13245 ts_tablet_manager.cc:1398] T f63e8e12fd3b453a96413f1fc769fb9f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:04.903479 15104 raft_consensus.cc:482] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.903643 15104 raft_consensus.cc:504] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.903908 13244 tablet_bootstrap.cc:492] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.904253 13244 ts_tablet_manager.cc:1367] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240506 19:57:04.904484 15104 leader_election.cc:288] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.907132 13244 raft_consensus.cc:348] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.907640 13244 raft_consensus.cc:374] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.907851 13244 raft_consensus.cc:729] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.908352 13244 consensus_queue.cc:260] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.910149 13244 ts_tablet_manager.cc:1398] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240506 19:57:04.915197 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02577dc6c2049a1abb47ed0fe729e18" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.915411 12988 raft_consensus.cc:2454] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:04.915992 13057 leader_election.cc:302] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters:
I20240506 19:57:04.916366 15104 raft_consensus.cc:2781] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240506 19:57:04.916397 15355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:57:04.916450 15104 raft_consensus.cc:482] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:04.916509 15104 raft_consensus.cc:3037] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.919037 15104 raft_consensus.cc:504] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.920192 15268 raft_consensus.cc:482] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.920331 15268 raft_consensus.cc:504] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.920944 15268 leader_election.cc:288] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:04.920944 15104 leader_election.cc:288] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:04.921733 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.923635 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02577dc6c2049a1abb47ed0fe729e18" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:04.924528 13058 leader_election.cc:341] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.925508 14982 server_base.cc:1034] running on GCE node
I20240506 19:57:04.925995 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02577dc6c2049a1abb47ed0fe729e18" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:04.926504 13058 leader_election.cc:341] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:04.928856 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02577dc6c2049a1abb47ed0fe729e18" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:04.929045 12988 raft_consensus.cc:3037] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.930402 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a568db06324a4fd1b58a29ef0831eb71" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.932760 12781 leader_election.cc:302] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.933164 15268 raft_consensus.cc:2726] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:04.933781 13249 tablet_bootstrap.cc:492] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.933913 13249 ts_tablet_manager.cc:1367] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.003s
I20240506 19:57:04.934621 13249 raft_consensus.cc:348] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.934773 13249 raft_consensus.cc:374] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.934814 13249 raft_consensus.cc:729] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.934952 15268 raft_consensus.cc:482] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:04.935091 15268 raft_consensus.cc:504] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.934986 13249 consensus_queue.cc:260] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.935699 15268 leader_election.cc:288] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:04.935731 13249 ts_tablet_manager.cc:1398] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:04.936012 13249 tablet_bootstrap.cc:492] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.936141 12988 raft_consensus.cc:2454] T e02577dc6c2049a1abb47ed0fe729e18 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:04.936761 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9f32618b64ffa92b5a868081c36a6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:04.937901 13057 leader_election.cc:302] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:04.938406 15104 raft_consensus.cc:2781] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:04.939334 13244 tablet_bootstrap.cc:492] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.939273 15104 raft_consensus.cc:686] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:04.941046 15104 consensus_queue.cc:237] T e02577dc6c2049a1abb47ed0fe729e18 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:04.943687 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d9f32618b64ffa92b5a868081c36a6" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:04.944389 13249 tablet_bootstrap.cc:654] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.944329 12781 leader_election.cc:302] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:04.944763 15268 raft_consensus.cc:2726] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 19:57:04.949846 15349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 19:57:04.951476 14982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 19:57:04.951761 13244 tablet_bootstrap.cc:654] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.960672 14982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 19:57:04.962152 14982 hybrid_clock.cc:648] HybridClock initialized: now 1715025424962022 us; error 71 us; skew 500 ppm
I20240506 19:57:04.962731 14982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 19:57:04.963217 13249 tablet_bootstrap.cc:492] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:04.963371 13249 ts_tablet_manager.cc:1367] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20240506 19:57:04.964184 13249 raft_consensus.cc:348] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.964407 13249 raft_consensus.cc:374] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.964468 13249 raft_consensus.cc:729] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.964702 13249 consensus_queue.cc:260] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:04.965242 13249 ts_tablet_manager.cc:1398] T f63e8e12fd3b453a96413f1fc769fb9f P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:04.972975 13245 tablet_bootstrap.cc:492] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:04.972635 14859 tablet_loader.cc:96] loaded metadata for tablet 6ec8883bfddf40058e09478e7fa5701b (table default.Table_22d4c2a939a34f7c9a0bbd4eb808d058 [id=531e45af97aa43639c67007d553b70c9])
I20240506 19:57:04.974689 13245 tablet_bootstrap.cc:654] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.979069 13249 tablet_bootstrap.cc:492] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:04.980733 13249 tablet_bootstrap.cc:654] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:04.982107 13244 tablet_bootstrap.cc:492] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:04.982264 13244 ts_tablet_manager.cc:1367] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240506 19:57:04.983088 13244 raft_consensus.cc:348] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.983302 13244 raft_consensus.cc:374] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:04.983366 13244 raft_consensus.cc:729] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:04.983575 13244 consensus_queue.cc:260] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:04.987941 13244 ts_tablet_manager.cc:1398] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240506 19:57:04.988245 12847 raft_consensus.cc:3037] T 9fed2f12eda54be0879b1e9c33ca3e31 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:04.988250 13244 tablet_bootstrap.cc:492] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:04.996457 13244 tablet_bootstrap.cc:654] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.010004 12848 raft_consensus.cc:3037] T 55d9f32618b64ffa92b5a868081c36a6 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.015326 13245 tablet_bootstrap.cc:492] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.015491 13245 ts_tablet_manager.cc:1367] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.008s
I20240506 19:57:05.016301 13245 raft_consensus.cc:348] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.016546 13245 raft_consensus.cc:374] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.016613 13245 raft_consensus.cc:729] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.016858 13244 tablet_bootstrap.cc:492] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.017014 13244 ts_tablet_manager.cc:1367] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20240506 19:57:05.016847 13245 consensus_queue.cc:260] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.017460 13245 ts_tablet_manager.cc:1398] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240506 19:57:05.017777 13244 raft_consensus.cc:348] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.017976 13244 raft_consensus.cc:374] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.018025 13244 raft_consensus.cc:729] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.018049 13245 tablet_bootstrap.cc:492] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.018227 13244 consensus_queue.cc:260] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.018765 13244 ts_tablet_manager.cc:1398] T 08535fa1600b41ff919fefa93cfc4c78 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:05.019032 13244 tablet_bootstrap.cc:492] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.019896 13245 tablet_bootstrap.cc:654] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.020622 13244 tablet_bootstrap.cc:654] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.022344 15268 raft_consensus.cc:482] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.022470 15268 raft_consensus.cc:504] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.023033 15268 leader_election.cc:288] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.023998 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4ad903e75547fa829b82cc722fef1e" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.026443 13249 tablet_bootstrap.cc:492] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:05.026830 13249 ts_tablet_manager.cc:1367] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.006s
I20240506 19:57:05.028836 13249 raft_consensus.cc:348] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.029436 13249 raft_consensus.cc:374] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.029640 13249 raft_consensus.cc:729] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.030158 13249 consensus_queue.cc:260] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.034088 13245 tablet_bootstrap.cc:492] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.034247 13245 ts_tablet_manager.cc:1367] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20240506 19:57:05.035054 13245 raft_consensus.cc:348] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.035240 13245 raft_consensus.cc:374] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.035300 13245 raft_consensus.cc:729] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.008155 14859 tablet_loader.cc:96] loaded metadata for tablet 6f4e97d10ae54e558920d116d8fa0ba5 (table default.Table_23747e8c1b314e4eaa4566b557389261 [id=a8b5f55150a54dcf8a19dc9e1153ea87])
I20240506 19:57:05.035511 13245 consensus_queue.cc:260] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.036027 13245 ts_tablet_manager.cc:1398] T 8a45a853e310473d844a423fbefc553a P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.036293 13245 tablet_bootstrap.cc:492] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.037214 13244 tablet_bootstrap.cc:492] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.037380 13244 ts_tablet_manager.cc:1367] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20240506 19:57:05.038130 13244 raft_consensus.cc:348] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.038352 13244 raft_consensus.cc:374] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.038408 13244 raft_consensus.cc:729] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.038442 13245 tablet_bootstrap.cc:654] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.038591 13244 consensus_queue.cc:260] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.039330 13244 ts_tablet_manager.cc:1398] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:05.038815 13249 ts_tablet_manager.cc:1398] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240506 19:57:05.039587 13244 tablet_bootstrap.cc:492] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.042279 14859 tablet_loader.cc:96] loaded metadata for tablet 6fb8bb09098d401da62ad737ffb697ad (table default.Table_223163dff40648e9824d80b4ca3a8894 [id=65fbfa6971874ba9a9f388961010594e])
I20240506 19:57:05.042261 13249 tablet_bootstrap.cc:492] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:05.044973 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4ad903e75547fa829b82cc722fef1e" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.047772 13244 tablet_bootstrap.cc:654] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.053218 13249 tablet_bootstrap.cc:654] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.056242 12781 leader_election.cc:302] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:05.056687 15268 raft_consensus.cc:2726] T 4b4ad903e75547fa829b82cc722fef1e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:05.057142 13249 tablet_bootstrap.cc:492] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:05.057300 13249 ts_tablet_manager.cc:1367] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240506 19:57:05.059274 14859 tablet_loader.cc:96] loaded metadata for tablet 6fea74d8996d4f238baafaa134d06691 (table default.Table_42386049e25d4c309be88fb22d88bc71 [id=979a99794a424140b5069524b475d849])
I20240506 19:57:05.059958 14859 tablet_loader.cc:96] loaded metadata for tablet 7037cab8dce1415ca11badeb693d9bec (table default.Table_e1bc856560494ef0ac2a3d693b45a7c0 [id=3af64f2f2b324a43add1c324664a45c1])
I20240506 19:57:05.060652 14859 tablet_loader.cc:96] loaded metadata for tablet 70b6a622cd9e483ca707440b4759e48f (table default.Table_f00049dcfdce4881891eed1bf5951d5b [id=f2b74839f0a0480f8d37f0d11f3743ea])
I20240506 19:57:05.061205 14859 tablet_loader.cc:96] loaded metadata for tablet 70e43e0a20cf4f05a7bc69278a83d1e3 (table default.Table_670635380d484804bfa6a7c94116e631 [id=f901d50539cf46869338d76c67dfcb4a])
I20240506 19:57:05.062024 14859 tablet_loader.cc:96] loaded metadata for tablet 70faf0c82cc84bc7befdfdea22be8a15 (table default.Table_d630c009c25b457388c1041305861480 [id=0c5aabe04b194b59b5b6adcd7389744a])
I20240506 19:57:05.062435 13245 tablet_bootstrap.cc:492] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.062583 13245 ts_tablet_manager.cc:1367] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240506 19:57:05.063414 13245 raft_consensus.cc:348] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.067968 13249 raft_consensus.cc:348] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.068197 13249 raft_consensus.cc:374] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.068248 13249 raft_consensus.cc:729] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.068440 13249 consensus_queue.cc:260] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.069010 13249 ts_tablet_manager.cc:1398] T dc206b31d3da458cb944d5f00e23d702 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20240506 19:57:05.076503 13244 tablet_bootstrap.cc:492] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.076658 13244 ts_tablet_manager.cc:1367] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s
I20240506 19:57:05.077466 13244 raft_consensus.cc:348] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.077704 13244 raft_consensus.cc:374] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.077764 13244 raft_consensus.cc:729] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.077970 13244 consensus_queue.cc:260] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.078518 13244 ts_tablet_manager.cc:1398] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:05.078802 13244 tablet_bootstrap.cc:492] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.081676 13244 tablet_bootstrap.cc:654] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.089771 12848 raft_consensus.cc:3037] T 1b89fb24d94042e1bfa772265a991680 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.093677 13244 tablet_bootstrap.cc:492] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.093823 13244 ts_tablet_manager.cc:1367] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20240506 19:57:05.094657 13244 raft_consensus.cc:348] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.094854 13244 raft_consensus.cc:374] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.094909 13244 raft_consensus.cc:729] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.094945 13245 raft_consensus.cc:374] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.095052 13245 raft_consensus.cc:729] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.095090 13244 consensus_queue.cc:260] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.095268 13245 consensus_queue.cc:260] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.095724 13244 ts_tablet_manager.cc:1398] T ea7e073296ce46e786362aaed96eb4dd P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:05.095863 13245 ts_tablet_manager.cc:1398] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.033s user 0.012s sys 0.004s
I20240506 19:57:05.095985 13244 tablet_bootstrap.cc:492] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.097617 13244 tablet_bootstrap.cc:654] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.099123 13249 tablet_bootstrap.cc:492] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:05.107664 13244 tablet_bootstrap.cc:492] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.107811 13244 ts_tablet_manager.cc:1367] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240506 19:57:05.108577 13244 raft_consensus.cc:348] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.108805 13244 raft_consensus.cc:374] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.108870 13244 raft_consensus.cc:729] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.109126 15104 raft_consensus.cc:482] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.109092 13244 consensus_queue.cc:260] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.109277 15104 raft_consensus.cc:504] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.109763 13244 ts_tablet_manager.cc:1398] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:05.109876 15104 leader_election.cc:288] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.110018 13244 tablet_bootstrap.cc:492] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
W20240506 19:57:05.110392 14081 leader_election.cc:334] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 27d4482cd59a4e5687992b1f8a177373 (127.12.48.253:36819): Timed out: RequestConsensusVote RPC to 127.12.48.253:36819 timed out after 0.682s (SENT)
I20240506 19:57:05.110530 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97f390a786034cddbff757779037f861; no voters: 190a2eabb30d4154960a44eb50a8a87d, 27d4482cd59a4e5687992b1f8a177373
I20240506 19:57:05.111729 13244 tablet_bootstrap.cc:654] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.111748 13245 tablet_bootstrap.cc:492] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.111783 12847 raft_consensus.cc:3037] T 5fe8d4891c484627ac729971ff7aad3e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.113041 13245 tablet_bootstrap.cc:654] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.115242 14910 raft_consensus.cc:482] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.115401 14910 raft_consensus.cc:504] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.116058 14910 leader_election.cc:288] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535)
I20240506 19:57:05.118028 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ece0bd3a3b47ef88a0b4910f0efae9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.122452 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee1a4a3f060407ca9c33467474e4489" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:05.124120 13058 leader_election.cc:341] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.125852 13244 tablet_bootstrap.cc:492] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.125887 14982 webserver.cc:460] Webserver started at http://127.12.48.254:40249/ using document root <none> and password file <none>
I20240506 19:57:05.125998 13244 ts_tablet_manager.cc:1367] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240506 19:57:05.126819 13244 raft_consensus.cc:348] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.127393 13244 raft_consensus.cc:374] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.127475 13244 raft_consensus.cc:729] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.127696 13244 consensus_queue.cc:260] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.128232 13244 ts_tablet_manager.cc:1398] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 19:57:05.128501 13244 tablet_bootstrap.cc:492] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.129678 15375 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 97f390a786034cddbff757779037f861 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 19:57:05.130158 13244 tablet_bootstrap.cc:654] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.131443 14982 fs_manager.cc:359] Metadata directory not provided
I20240506 19:57:05.131615 14982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 19:57:05.137615 13244 tablet_bootstrap.cc:492] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.137738 13244 ts_tablet_manager.cc:1367] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240506 19:57:05.138435 13244 raft_consensus.cc:348] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.138643 13244 raft_consensus.cc:374] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.138707 13244 raft_consensus.cc:729] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.138896 13244 consensus_queue.cc:260] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.139406 13244 ts_tablet_manager.cc:1398] T 1e22e5a015f74526b49363bc789cb4ed P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:05.139714 13244 tablet_bootstrap.cc:492] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.141371 13244 tablet_bootstrap.cc:654] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.142978 15103 raft_consensus.cc:482] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.143150 15103 raft_consensus.cc:504] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.144663 13249 tablet_bootstrap.cc:654] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.144649 15103 leader_election.cc:288] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.142302 12847 raft_consensus.cc:3037] T 028055b9df1f4667bd2e54909690e327 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.147688 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca23d6f0a9e0445cbd4d0e708c8de2d3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:05.149070 13058 leader_election.cc:341] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.151820 13244 tablet_bootstrap.cc:492] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.151963 13244 ts_tablet_manager.cc:1367] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240506 19:57:05.152734 13244 raft_consensus.cc:348] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.152927 13244 raft_consensus.cc:374] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.152989 13244 raft_consensus.cc:729] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.153190 13244 consensus_queue.cc:260] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.153237 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee1a4a3f060407ca9c33467474e4489" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.153782 13244 ts_tablet_manager.cc:1398] T 7a396afff3b243ef84cb3dc3836fcdfd P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:05.154045 13244 tablet_bootstrap.cc:492] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.153981 12988 raft_consensus.cc:2454] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:05.155634 13244 tablet_bootstrap.cc:654] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.164175 13244 tablet_bootstrap.cc:492] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.164407 13244 ts_tablet_manager.cc:1367] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240506 19:57:05.165210 13244 raft_consensus.cc:348] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.165488 13244 raft_consensus.cc:374] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.165685 13244 raft_consensus.cc:729] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.165923 13244 consensus_queue.cc:260] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.166474 13244 ts_tablet_manager.cc:1398] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 19:57:05.166792 13244 tablet_bootstrap.cc:492] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.168418 13244 tablet_bootstrap.cc:654] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.168373 14859 tablet_loader.cc:96] loaded metadata for tablet 71924feef80a43b19828756fe7ecba56 (table default.Table_faeb0284bba8473ebdd24f032788bdf7 [id=cfe6ab0aa2c14d0b8b15653edde3dae4])
I20240506 19:57:05.169265 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ece0bd3a3b47ef88a0b4910f0efae9" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.173662 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca23d6f0a9e0445cbd4d0e708c8de2d3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.173898 12988 raft_consensus.cc:2454] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:05.179980 13057 leader_election.cc:302] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:05.180459 15103 raft_consensus.cc:2781] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:05.180569 15103 raft_consensus.cc:482] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:05.180686 15103 raft_consensus.cc:3037] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.183779 15103 raft_consensus.cc:504] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.184576 15103 leader_election.cc:288] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.185834 13249 tablet_bootstrap.cc:492] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:05.186082 13249 ts_tablet_manager.cc:1367] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.006s
I20240506 19:57:05.187335 13249 raft_consensus.cc:348] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.191071 13249 raft_consensus.cc:374] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.191417 13249 raft_consensus.cc:729] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.191692 13249 consensus_queue.cc:260] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.192298 13249 ts_tablet_manager.cc:1398] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240506 19:57:05.196926 12781 leader_election.cc:302] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:05.197415 14910 raft_consensus.cc:2726] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:05.199725 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee1a4a3f060407ca9c33467474e4489" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:05.201402 13058 leader_election.cc:341] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.205706 14859 tablet_loader.cc:96] loaded metadata for tablet 71e4cd127e494acd8b7dffd87cb6bfa0 (table default.Table_36502249055f419692c8fe55e7f7d2ea [id=59360d532f304a3bbab1c482d0ba8c03])
I20240506 19:57:05.206761 12848 raft_consensus.cc:3037] T a03f574aec984b83ab596e92a6f323fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.210287 15103 raft_consensus.cc:482] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.210610 15103 raft_consensus.cc:504] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.211728 15103 leader_election.cc:288] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.212828 14859 tablet_loader.cc:96] loaded metadata for tablet 7238f1b2905b4b8bb0e2b764ea838c56 (table default.Table_eb33ab4810b94486b70e278e548b02ba [id=bea1ca1264dc4a44a9c2e91d52ae2a81])
I20240506 19:57:05.220597 13057 leader_election.cc:302] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:05.225654 15104 raft_consensus.cc:2781] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:05.225786 15104 raft_consensus.cc:482] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:05.225869 15104 raft_consensus.cc:3037] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.229665 15104 raft_consensus.cc:504] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.230962 15104 leader_election.cc:288] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.236997 13249 tablet_bootstrap.cc:492] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:05.240895 14910 raft_consensus.cc:482] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.241079 14910 raft_consensus.cc:504] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.241994 14910 leader_election.cc:288] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
W20240506 19:57:05.243455 12783 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: 20aa06290b9a4814a006c423ccd8fc46
I20240506 19:57:05.248209 12847 raft_consensus.cc:3037] T 01c32eba2031446ab419cac64068068e P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:05.255009 13058 consensus_peers.cc:485] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:05.256894 12846 raft_consensus.cc:3037] T d0aae034f52645ba96827923f4fa305c P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.266425 13244 tablet_bootstrap.cc:492] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.266721 13244 ts_tablet_manager.cc:1367] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.022s
I20240506 19:57:05.266306 13249 tablet_bootstrap.cc:654] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.266954 12843 raft_consensus.cc:3037] T ffbb87878d06417cbabf3c218728dbdc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.267623 13244 raft_consensus.cc:348] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.267817 13244 raft_consensus.cc:374] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.267866 13244 raft_consensus.cc:729] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.268050 13244 consensus_queue.cc:260] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.268582 13244 ts_tablet_manager.cc:1398] T 4dcc036a20824736b16815bcb47645c4 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.268855 13244 tablet_bootstrap.cc:492] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.270664 13244 tablet_bootstrap.cc:654] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
W20240506 19:57:05.283845 13442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240506 19:57:05.290975 14910 raft_consensus.cc:482] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.291168 14910 raft_consensus.cc:504] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.292109 14910 leader_election.cc:288] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.292805 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
W20240506 19:57:05.293687 13442 consensus_peers.cc:485] T 00000000000000000000000000000000 P 27d4482cd59a4e5687992b1f8a177373 -> Peer 190a2eabb30d4154960a44eb50a8a87d (127.12.48.254:33495): Couldn't send request to peer 190a2eabb30d4154960a44eb50a8a87d. Status: Network error: Client connection negotiation failed: client connection to 127.12.48.254:33495: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240506 19:57:05.295226 15009 raft_consensus.cc:482] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.295395 15009 raft_consensus.cc:504] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.296001 15009 leader_election.cc:288] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.296790 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d37bb1c2144c4fa71aab6113b9f06e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:05.298749 12921 leader_election.cc:341] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.303351 14859 tablet_loader.cc:96] loaded metadata for tablet 7300ca61d4474d2183f4281909400897 (table default.Table_111e3f9425804b4ca0bf599cc9377867 [id=8026f48a5250456a8cea07bb6446b845])
I20240506 19:57:05.303982 14859 tablet_loader.cc:96] loaded metadata for tablet 7380f259c68d4d7c89938a548533b354 (table default.Table_efa08afa30fd48009749a76c90bd6725 [id=0df30d7b42ff4869ad37a24a82300280])
I20240506 19:57:05.305636 14859 tablet_loader.cc:96] loaded metadata for tablet 73c4aba81db7470f87728def61bbf71c (table default.Table_fc32d4e070f24c35a86b7bfb8cbfd7e6 [id=0090970ea0b54ee4a872766d8145bd6f])
I20240506 19:57:05.306748 14859 tablet_loader.cc:96] loaded metadata for tablet 7446b0633a484cfba10f44a69dcc6969 (table default.Table_9765ef1ea1284910aea85d1d0de68bb3 [id=ed62c20271fe4b58be6d1d66643da8a1])
I20240506 19:57:05.311736 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f45e9fa2cb945d2a04fa716d1ab33b4" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.311738 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee1a4a3f060407ca9c33467474e4489" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:05.312664 12983 raft_consensus.cc:3037] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.315289 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d37bb1c2144c4fa71aab6113b9f06e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.315518 13114 raft_consensus.cc:2454] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:05.316218 12983 raft_consensus.cc:2454] T 0ee1a4a3f060407ca9c33467474e4489 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:05.323715 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca23d6f0a9e0445cbd4d0e708c8de2d3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:05.324350 12842 raft_consensus.cc:3037] T a568db06324a4fd1b58a29ef0831eb71 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.328933 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c19567619b64ee0a8dd637d7f69ce3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.334510 13245 tablet_bootstrap.cc:492] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.334677 13245 ts_tablet_manager.cc:1367] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.223s user 0.042s sys 0.054s
I20240506 19:57:05.335310 13245 raft_consensus.cc:348] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.335510 13245 raft_consensus.cc:374] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.335563 13245 raft_consensus.cc:729] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.335755 13245 consensus_queue.cc:260] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.336266 13245 ts_tablet_manager.cc:1398] T 6097fe368a894f648b097912e16e361f P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.336524 13245 tablet_bootstrap.cc:492] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.338001 13245 tablet_bootstrap.cc:654] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.346263 12983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c19567619b64ee0a8dd637d7f69ce3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.346382 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca23d6f0a9e0445cbd4d0e708c8de2d3" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:05.346531 12987 raft_consensus.cc:3037] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
W20240506 19:57:05.346974 12781 leader_election.cc:341] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527): Not found: Tablet not found: 20aa06290b9a4814a006c423ccd8fc46
I20240506 19:57:05.347154 12781 leader_election.cc:302] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:05.347754 14910 raft_consensus.cc:2726] T 20aa06290b9a4814a006c423ccd8fc46 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:05.348471 15248 raft_consensus.cc:482] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.348673 15248 raft_consensus.cc:504] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.349335 12987 raft_consensus.cc:2454] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 1.
I20240506 19:57:05.346501 12983 raft_consensus.cc:2454] T 9c19567619b64ee0a8dd637d7f69ce3e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:05.349854 15248 leader_election.cc:288] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.352941 14910 raft_consensus.cc:482] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.353060 14910 raft_consensus.cc:504] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.354162 14910 leader_election.cc:288] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.354297 13249 tablet_bootstrap.cc:492] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:05.354640 13245 tablet_bootstrap.cc:492] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.354602 13249 ts_tablet_manager.cc:1367] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.006s
I20240506 19:57:05.354771 13245 ts_tablet_manager.cc:1367] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
W20240506 19:57:05.355484 12783 leader_election.cc:341] T f8ffb88e0a0c4125b04f4d8d328da3f5 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535): Not found: Tablet not found: f8ffb88e0a0c4125b04f4d8d328da3f5
I20240506 19:57:05.355413 13245 raft_consensus.cc:348] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.355590 13245 raft_consensus.cc:374] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.355666 13245 raft_consensus.cc:729] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.355854 13245 consensus_queue.cc:260] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.356355 13245 ts_tablet_manager.cc:1398] T dc206b31d3da458cb944d5f00e23d702 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.356591 13245 tablet_bootstrap.cc:492] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.356302 13249 raft_consensus.cc:348] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.356738 13249 raft_consensus.cc:374] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.356892 13249 raft_consensus.cc:729] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.358158 13245 tablet_bootstrap.cc:654] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.358876 15010 raft_consensus.cc:482] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.359026 15010 raft_consensus.cc:504] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.359673 15010 leader_election.cc:288] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.358558 13249 consensus_queue.cc:260] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.361342 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902d43d15d334939bddb254ae04ca174" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.361759 13245 tablet_bootstrap.cc:492] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.361876 13245 ts_tablet_manager.cc:1367] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240506 19:57:05.362474 13245 raft_consensus.cc:348] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.362658 13245 raft_consensus.cc:374] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.362710 13245 raft_consensus.cc:729] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.362875 13245 consensus_queue.cc:260] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
W20240506 19:57:05.363107 12921 leader_election.cc:341] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.363356 13245 ts_tablet_manager.cc:1398] T dc2c509a06554b9db6d5e0be97e159db P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 19:57:05.363633 13245 tablet_bootstrap.cc:492] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.365096 13245 tablet_bootstrap.cc:654] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.367043 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19db9cff74843e087fd82db2475c29c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:05.367578 12921 leader_election.cc:341] T d19db9cff74843e087fd82db2475c29c P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.368685 13249 ts_tablet_manager.cc:1398] T dc2c509a06554b9db6d5e0be97e159db P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.014s user 0.002s sys 0.002s
I20240506 19:57:05.376370 13249 tablet_bootstrap.cc:492] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:05.377173 13245 tablet_bootstrap.cc:492] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.377341 13245 ts_tablet_manager.cc:1367] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240506 19:57:05.378080 13245 raft_consensus.cc:348] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.378259 13245 raft_consensus.cc:374] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.378314 13245 raft_consensus.cc:729] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.378484 13245 consensus_queue.cc:260] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.378995 13245 ts_tablet_manager.cc:1398] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.382627 12781 leader_election.cc:302] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef18e88b87064ed4b6c20f9325ebcc33; no voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d
I20240506 19:57:05.383097 14910 raft_consensus.cc:2726] T 2f45e9fa2cb945d2a04fa716d1ab33b4 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 19:57:05.386271 15104 consensus_queue.cc:1035] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:05.386848 15104 consensus_queue.cc:1035] T 0b899244c35b4bbe9e916c778111f0c0 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 19:57:05.388602 13058 leader_election.cc:341] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:05.388841 13058 leader_election.cc:341] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:05.389058 13058 consensus_peers.cc:485] T c5f3def738fb4cb195d08e48d7d2b293 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:05.389271 13058 consensus_peers.cc:485] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 19:57:05.391093 13058 consensus_peers.cc:485] T cc22261ab2034e818834ba97c2760dad P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:05.392113 13245 tablet_bootstrap.cc:492] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.392021 13057 leader_election.cc:302] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:05.393695 13245 tablet_bootstrap.cc:654] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.395676 15104 raft_consensus.cc:2781] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:05.396291 15104 raft_consensus.cc:686] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:05.396584 15104 consensus_queue.cc:237] T 0ee1a4a3f060407ca9c33467474e4489 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.398149 13249 tablet_bootstrap.cc:654] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.404649 14915 raft_consensus.cc:482] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.404800 14915 raft_consensus.cc:504] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.405755 14915 leader_election.cc:288] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.405869 14915 raft_consensus.cc:482] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.405949 14915 raft_consensus.cc:504] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.406420 14915 leader_election.cc:288] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.412866 14910 raft_consensus.cc:482] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.413120 14910 raft_consensus.cc:504] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.414053 14910 leader_election.cc:288] T f078559df48f4d23869ddcb4b3fca9fc P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.414996 13245 tablet_bootstrap.cc:492] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.415139 13245 ts_tablet_manager.cc:1367] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.002s
I20240506 19:57:05.415150 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f078559df48f4d23869ddcb4b3fca9fc" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.415916 13245 raft_consensus.cc:348] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.416101 13245 raft_consensus.cc:374] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.416162 13245 raft_consensus.cc:729] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.416357 13245 consensus_queue.cc:260] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.416862 13245 ts_tablet_manager.cc:1398] T 0bc56299ed854d6c8ef201127692fd60 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.417135 13245 tablet_bootstrap.cc:492] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.418797 13245 tablet_bootstrap.cc:654] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.422761 12845 raft_consensus.cc:3037] T c8ece0bd3a3b47ef88a0b4910f0efae9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.430382 13245 tablet_bootstrap.cc:492] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.430545 13245 ts_tablet_manager.cc:1367] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240506 19:57:05.431253 13245 raft_consensus.cc:348] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.431452 13245 raft_consensus.cc:374] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.431501 13245 raft_consensus.cc:729] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.431705 13245 consensus_queue.cc:260] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.432186 13245 ts_tablet_manager.cc:1398] T 7bc30f4690b24c5ea8ab2f7a26122992 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.432469 13245 tablet_bootstrap.cc:492] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.434137 13245 tablet_bootstrap.cc:654] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.435822 13245 tablet_bootstrap.cc:492] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.435947 13245 ts_tablet_manager.cc:1367] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240506 19:57:05.436616 13245 raft_consensus.cc:348] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.436789 13245 raft_consensus.cc:374] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.436838 13245 raft_consensus.cc:729] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.437029 13245 consensus_queue.cc:260] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.437536 13245 ts_tablet_manager.cc:1398] T 3f7984f6e0fe485fa0c15fd13c3f428e P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:05.437814 13245 tablet_bootstrap.cc:492] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.439332 13245 tablet_bootstrap.cc:654] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.441359 15103 raft_consensus.cc:482] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.441607 15103 raft_consensus.cc:504] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.442688 15103 leader_election.cc:288] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.447234 13245 tablet_bootstrap.cc:492] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.447382 13245 ts_tablet_manager.cc:1367] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240506 19:57:05.448815 13245 raft_consensus.cc:348] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.449041 13245 raft_consensus.cc:374] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.449124 13245 raft_consensus.cc:729] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.449362 13245 consensus_queue.cc:260] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.449899 13245 ts_tablet_manager.cc:1398] T a8448ad8e9384edba7202d0bdce516a4 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 19:57:05.450213 13245 tablet_bootstrap.cc:492] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Bootstrap starting.
I20240506 19:57:05.451906 13245 tablet_bootstrap.cc:654] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.457118 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902d43d15d334939bddb254ae04ca174" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.457803 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8f5301f524e4e8910ce61837a1dd4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.461519 15010 raft_consensus.cc:482] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.461733 15010 raft_consensus.cc:504] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.462679 15010 leader_election.cc:288] T c652feb405ef4f1cab8ee602e9d3eb18 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.465153 15009 consensus_queue.cc:1035] T ee69d8e09c2f4c898ce547ec396ca0e6 P 059a5c4a3f7e49ef95af4b8341702745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 19:57:05.466557 12922 leader_election.cc:302] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:05.466997 15009 raft_consensus.cc:2781] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:05.467134 15009 raft_consensus.cc:482] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:05.467244 15009 raft_consensus.cc:3037] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.471920 15009 raft_consensus.cc:504] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.475762 15009 leader_election.cc:288] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.475968 15104 raft_consensus.cc:482] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.476112 15104 raft_consensus.cc:504] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.476738 15104 leader_election.cc:288] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.478431 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d37bb1c2144c4fa71aab6113b9f06e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
W20240506 19:57:05.479135 12921 leader_election.cc:341] T e6d37bb1c2144c4fa71aab6113b9f06e P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.485620 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c652feb405ef4f1cab8ee602e9d3eb18" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.487964 14859 tablet_loader.cc:96] loaded metadata for tablet 74efea80334249f1b4ac504de7ca4fa9 (table default.Table_922f89e002cf480a9fca0e23f0eeb801 [id=ef089914d9514c29b2e3341c27e1a084])
I20240506 19:57:05.488559 14859 tablet_loader.cc:96] loaded metadata for tablet 7525faf76e174113aa00d5b53aa6ea22 (table default.Table_fd50aaa5e06941b0b3fbb1d2e4966cbb [id=5d6532bd90404984afd9c8d0aa099990])
I20240506 19:57:05.490248 14859 tablet_loader.cc:96] loaded metadata for tablet 76e6f7555dfa4122ac00bae7b9e733fe (table default.Table_dbebd827d8ed405bae594e5adc6c14a9 [id=9006912cbc764b9a8f90a11d963cf02a])
I20240506 19:57:05.499485 14910 raft_consensus.cc:482] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.499682 14910 raft_consensus.cc:504] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } }
I20240506 19:57:05.501116 14910 leader_election.cc:288] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527)
I20240506 19:57:05.502007 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744864043f9347e0a13f8c2a7d6b147f" candidate_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.503438 12846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a18ad478630e491cab700ac0c42a5ede" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.504045 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad7076ed00f4aec9cb92ba7860932b6" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
W20240506 19:57:05.505625 13058 leader_election.cc:341] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:05.505990 13058 leader_election.cc:341] T 4ea8f5301f524e4e8910ce61837a1dd4 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.520606 15009 raft_consensus.cc:482] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.520844 15009 raft_consensus.cc:504] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.522725 13057 leader_election.cc:302] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:05.523640 14915 raft_consensus.cc:2781] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 19:57:05.523790 14915 raft_consensus.cc:686] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Running, Role: LEADER
I20240506 19:57:05.524102 14915 consensus_queue.cc:237] T ca23d6f0a9e0445cbd4d0e708c8de2d3 P 8a9c37db9d4f41cd93820a1c81f48d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.534531 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19db9cff74843e087fd82db2475c29c" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.534837 13114 raft_consensus.cc:2454] T d19db9cff74843e087fd82db2475c29c P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:05.554256 13057 leader_election.cc:302] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059a5c4a3f7e49ef95af4b8341702745, 8a9c37db9d4f41cd93820a1c81f48d4d; no voters: ef18e88b87064ed4b6c20f9325ebcc33
I20240506 19:57:05.555073 15104 raft_consensus.cc:2781] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 19:57:05.555234 15104 raft_consensus.cc:482] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 19:57:05.555377 15104 raft_consensus.cc:3037] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.559059 15104 raft_consensus.cc:504] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.563755 13244 tablet_bootstrap.cc:492] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.564244 13244 ts_tablet_manager.cc:1367] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.295s user 0.034s sys 0.038s
I20240506 19:57:05.566659 15009 leader_election.cc:288] T 65da8b15e0c6496bb86b47c49e348af0 P 059a5c4a3f7e49ef95af4b8341702745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9c37db9d4f41cd93820a1c81f48d4d (127.12.48.195:40535), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.566691 13244 raft_consensus.cc:348] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.567265 13244 raft_consensus.cc:374] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.567411 13244 raft_consensus.cc:729] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18e88b87064ed4b6c20f9325ebcc33, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.567977 13244 consensus_queue.cc:260] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.568994 12842 raft_consensus.cc:3037] T 744864043f9347e0a13f8c2a7d6b147f P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.569485 13244 ts_tablet_manager.cc:1398] T 7fd12a9cc3924f3eb8aec502e9fa2d47 P ef18e88b87064ed4b6c20f9325ebcc33: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240506 19:57:05.570603 13244 tablet_bootstrap.cc:492] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P ef18e88b87064ed4b6c20f9325ebcc33: Bootstrap starting.
I20240506 19:57:05.575032 13244 tablet_bootstrap.cc:654] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P ef18e88b87064ed4b6c20f9325ebcc33: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.577052 14982 fs_manager.cc:709] Time spent opening directory manager: real 0.374s user 0.005s sys 0.002s
I20240506 19:57:05.577591 14859 tablet_loader.cc:96] loaded metadata for tablet 76fb4d67510e413d9f23b021dae43b0f (table default.Table_aa2cfdcd14e74072be6f657fcb05127e [id=a5c3329d55274c708fa24ff378b7cea0])
I20240506 19:57:05.579883 14859 tablet_loader.cc:96] loaded metadata for tablet 778075705b0b4f00a3184f42b68d2323 (table default.Table_951df9b3f85e416893c271dc056e57ee [id=27ca3baadf324fbe92ecfe455e90428a])
I20240506 19:57:05.582712 14859 tablet_loader.cc:96] loaded metadata for tablet 779534d888ab4c41bae0d7fe5fffc390 (table default.Table_91e0d90fa1a442a2a442d06baf1ef73f [id=49164dc63a4a4b00b435277562ee5224])
I20240506 19:57:05.583174 14859 tablet_loader.cc:96] loaded metadata for tablet 78f503285b7348faabb5ea699787f088 (table default.Table_114d9e27c6d74425802c46e8610d3343 [id=efe4c73c778d4128b0ff1a51a985f5c1])
I20240506 19:57:05.583796 15104 leader_election.cc:288] T 9c19567619b64ee0a8dd637d7f69ce3e P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 election: Requested vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
I20240506 19:57:05.585402 14859 tablet_loader.cc:96] loaded metadata for tablet 797c3629f1b0413eb471cc899f54b32d (table default.Table_0f8f0a413eab460aa84362ac194994e1 [id=33dfa30213944d2883cefa6f0c6f9365])
I20240506 19:57:05.581094 14915 raft_consensus.cc:482] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.585731 14915 raft_consensus.cc:504] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.586881 14915 leader_election.cc:288] T 65da8b15e0c6496bb86b47c49e348af0 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059a5c4a3f7e49ef95af4b8341702745 (127.12.48.194:43527), ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475)
W20240506 19:57:05.561153 13058 consensus_peers.cc:485] T 8321457717fd4ad28ae1292d9e9a8669 P 8a9c37db9d4f41cd93820a1c81f48d4d -> Peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Couldn't send request to peer ef18e88b87064ed4b6c20f9325ebcc33. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240506 19:57:05.587263 14859 tablet_loader.cc:96] loaded metadata for tablet 79f643b5e89e470aa35f258109bbc22f (table default.Table_742be09ecdef4a4c9d11965a17109e81 [id=82062552076642299d2d7e511dd6bebf])
W20240506 19:57:05.587399 13058 leader_election.cc:341] T a18ad478630e491cab700ac0c42a5ede P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
W20240506 19:57:05.587699 13058 leader_election.cc:341] T 8ad7076ed00f4aec9cb92ba7860932b6 P 8a9c37db9d4f41cd93820a1c81f48d4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef18e88b87064ed4b6c20f9325ebcc33 (127.12.48.193:39475): Illegal state: must be running to vote when last-logged opid is not known
I20240506 19:57:05.588408 14859 tablet_loader.cc:96] loaded metadata for tablet 7a396afff3b243ef84cb3dc3836fcdfd (table default.Table_d01da29ce66546ab88b2eba38a69c4d1 [id=9d511b62ea4b4e66a721a3508f15540e])
I20240506 19:57:05.588860 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902d43d15d334939bddb254ae04ca174" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.589118 13114 raft_consensus.cc:2454] T 902d43d15d334939bddb254ae04ca174 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:05.598368 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c652feb405ef4f1cab8ee602e9d3eb18" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" is_pre_election: true
I20240506 19:57:05.598649 13249 tablet_bootstrap.cc:492] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745: No bootstrap required, opened a new log
I20240506 19:57:05.598630 13120 raft_consensus.cc:2454] T c652feb405ef4f1cab8ee602e9d3eb18 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 0.
I20240506 19:57:05.598791 13249 ts_tablet_manager.cc:1367] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745: Time spent bootstrapping tablet: real 0.227s user 0.014s sys 0.009s
I20240506 19:57:05.599479 13249 raft_consensus.cc:348] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.599684 13249 raft_consensus.cc:374] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.599740 13249 raft_consensus.cc:729] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a5c4a3f7e49ef95af4b8341702745, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.599946 13249 consensus_queue.cc:260] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } }
I20240506 19:57:05.600455 13249 ts_tablet_manager.cc:1398] T 3f7984f6e0fe485fa0c15fd13c3f428e P 059a5c4a3f7e49ef95af4b8341702745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 19:57:05.600736 13249 tablet_bootstrap.cc:492] T 8a45a853e310473d844a423fbefc553a P 059a5c4a3f7e49ef95af4b8341702745: Bootstrap starting.
I20240506 19:57:05.602546 13116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d37bb1c2144c4fa71aab6113b9f06e" candidate_uuid: "059a5c4a3f7e49ef95af4b8341702745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d"
I20240506 19:57:05.602737 13116 raft_consensus.cc:3037] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Advancing to term 1
I20240506 19:57:05.605741 13116 raft_consensus.cc:2454] T e6d37bb1c2144c4fa71aab6113b9f06e P 8a9c37db9d4f41cd93820a1c81f48d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059a5c4a3f7e49ef95af4b8341702745 in term 1.
I20240506 19:57:05.606369 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c19567619b64ee0a8dd637d7f69ce3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33"
I20240506 19:57:05.607187 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da8b15e0c6496bb86b47c49e348af0" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" is_pre_election: true
I20240506 19:57:05.611968 14859 tablet_loader.cc:96] loaded metadata for tablet 7a4a9693c9fb411db07847fede883920 (table default.Table_16db473ae5ed4bd0ab2c8795b490f5cf [id=0b3a486ff6d24c8795de3e53e83021a2])
I20240506 19:57:05.612675 13249 tablet_bootstrap.cc:654] T 8a45a853e310473d844a423fbefc553a P 059a5c4a3f7e49ef95af4b8341702745: Neither blocks nor log segments found. Creating new log.
I20240506 19:57:05.617197 14859 tablet_loader.cc:96] loaded metadata for tablet 7b803ae289344f88b604a2053881d59b (table default.Table_6013611882be454da8ec1ea823261f6a [id=d2ffd5634f2042b7a85cbe74d6ca1df5])
I20240506 19:57:05.618180 13245 tablet_bootstrap.cc:492] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d: No bootstrap required, opened a new log
I20240506 19:57:05.618427 13245 ts_tablet_manager.cc:1367] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent bootstrapping tablet: real 0.168s user 0.052s sys 0.071s
I20240506 19:57:05.619624 13245 raft_consensus.cc:348] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.619947 13245 raft_consensus.cc:374] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.620054 13245 raft_consensus.cc:729] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c37db9d4f41cd93820a1c81f48d4d, State: Initialized, Role: FOLLOWER
I20240506 19:57:05.620332 13245 consensus_queue.cc:260] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.628857 13245 ts_tablet_manager.cc:1398] T 8bd1b51c240a4b1aad65839679f7bba7 P 8a9c37db9d4f41cd93820a1c81f48d4d: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20240506 19:57:05.630162 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8f5301f524e4e8910ce61837a1dd4" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.630374 12986 raft_consensus.cc:2454] T 4ea8f5301f524e4e8910ce61837a1dd4 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:05.630445 13244 tablet_bootstrap.cc:492] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P ef18e88b87064ed4b6c20f9325ebcc33: No bootstrap required, opened a new log
I20240506 19:57:05.630870 13244 ts_tablet_manager.cc:1367] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P ef18e88b87064ed4b6c20f9325ebcc33: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.000s
I20240506 19:57:05.631546 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902d43d15d334939bddb254ae04ca174" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.631731 12987 raft_consensus.cc:2454] T 902d43d15d334939bddb254ae04ca174 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:05.632936 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad7076ed00f4aec9cb92ba7860932b6" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.633141 12986 raft_consensus.cc:2454] T 8ad7076ed00f4aec9cb92ba7860932b6 P 059a5c4a3f7e49ef95af4b8341702745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9c37db9d4f41cd93820a1c81f48d4d in term 0.
I20240506 19:57:05.635017 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a18ad478630e491cab700ac0c42a5ede" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745" is_pre_election: true
I20240506 19:57:05.634382 13244 raft_consensus.cc:348] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195" port: 40535 } } peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } }
I20240506 19:57:05.636327 13244 raft_consensus.cc:374] T 9f30a2ce53464d12b98c6a0c2fdeb4fa P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 19:57:05.636615 14910 raft_consensus.cc:482] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 19:57:05.636792 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c19567619b64ee0a8dd637d7f69ce3e" candidate_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059a5c4a3f7e49ef95af4b8341702745"
I20240506 19:57:05.636909 14910 raft_consensus.cc:504] T 44ce8749846847108edf5f554139cbd9 P ef18e88b87064ed4b6c20f9325ebcc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef18e88b87064ed4b6c20f9325ebcc33" member_type: VOTER last_known_addr { host: "127.12.48.193" port: 39475 } } peers { permanent_uuid: "059a5c4a3f7e49ef95af4b8341702745" member_type: VOTER last_known_addr { host: "127.12.48.194" port: 43527 } } peers { permanent_uuid: "8a9c37db9d4f41cd93820a1c81f48d4d" member_type: VOTER last_known_addr { host: "127.12.48.195"